Nov 02 21:56:59.960514 ip-10-0-131-57 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 21:57:00.385262 ip-10-0-131-57 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:00.385262 ip-10-0-131-57 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 21:57:00.385262 ip-10-0-131-57 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:00.385262 ip-10-0-131-57 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:00.390250 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390154    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.390250 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390248    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.390250 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390252    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.390250 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390255    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390259    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390262    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390265    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390268    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390270    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390275    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390279    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390283    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390287    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390290    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390308    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390311    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390314    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390316    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390319    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390322    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390324    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390327    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390329    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.390369 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390333    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390336    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390338    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390341    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390343    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390346    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390349    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390352    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390355    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390357    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390360    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390362    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390365    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390368    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390372    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390375    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390378    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390381    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390384    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390386    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.390854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390389    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390392    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390394    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390398    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390401    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390403    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390406    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390409    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390411    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390413    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390416    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390418    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390421    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390424    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390427    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390430    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390432    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390435    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390437    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.391595 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390440    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390443    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390445    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390448    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390450    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390453    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390456    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390459    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390461    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390464    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390467    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390470    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390472    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390475    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390477    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390480    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390482    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390485    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390488    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.392077 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390490    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390495    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390499    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390501    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390504    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390507    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390510    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390513    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390515    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390518    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390521    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390523    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390526    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390528    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390531    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390533    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390536    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390539    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390542    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.392565 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390546    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.393022 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390548    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.393022 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390551    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.393022 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390553    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.393022 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390556    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.393022 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.390558    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.394498 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394482    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.394498 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394498    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394502    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394505    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394508    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394511    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394514    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394516    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394519    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394522    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394525    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394527    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394530    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394532    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394535    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394538    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394540    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394543    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394545    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394548    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394550    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.394603 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394553    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394555    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394558    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394570    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394578    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394580    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394584    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394587    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394591    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394595    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394598    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394601    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394604    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394607    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394610    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394613    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394616    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394619    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394621    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.395089 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394624    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394627    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394629    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394632    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394634    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394636    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394639    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394641    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394644    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394646    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394649    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394652    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394654    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394657    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394659    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394663    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394666    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394669    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394671    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394674    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.395554 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394677    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394679    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394682    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394685    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394687    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394690    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394693    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394695    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394698    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394700    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394703    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394705    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394708    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394710    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394714    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394716    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394719    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394721    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394724    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.396032 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394727    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394729    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394731    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394734    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394737    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394739    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394742    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394745    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394748    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394751    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394753    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394756    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394758    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394761    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394763    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394766    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394768    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394771    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394773    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394776    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.396500 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394778    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394781    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394785    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394789    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394793    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394796    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.394799    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395331    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395336    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395339    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395342    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395345    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395348    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395351    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395354    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395356    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395360    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395364    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395366    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.396971 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395369    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395372    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395375    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395377    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395380    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395385    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395387    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395390    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395393    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395395    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395398    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395400    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395403    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395406    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395408    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395411    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395414    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395416    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.397432 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395419    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395421    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395424    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395426    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395429    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395431    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395437    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395439    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395442    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395445    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395449    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395451    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395454    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395456    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395459    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395462    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395464    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395467    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395469    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395472    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.397883 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395475    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395477    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395479    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395482    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395484    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395487    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395491    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395495    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395498    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395500    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395503    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395505    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395508    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395510    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395513    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395515    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395518    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395521    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395524    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.398410 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395527    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395529    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395532    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395535    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395538    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395541    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395544    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395546    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395549    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395552    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395554    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395557    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395560    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395563    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395565    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395568    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395570    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395573    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395575    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.398855 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395578    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395580    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395583    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395587    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395591    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395594    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395598    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395600    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395603    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395605    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395610    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395613    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395615    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395618    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395620    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.395623    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395728    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395735    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395742    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395747    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395752    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 21:57:00.399338 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395755    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395759    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395764    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395767    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395770    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395774    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395778    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395781    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395784    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395787    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395790    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395793    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395796    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395801    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395804    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395807    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395811    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395815    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395823    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395826    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395830    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395835    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395838    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395841    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 21:57:00.399824 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395844    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395847    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395850    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395855    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395859    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395862    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395865    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395868    2576 flags.go:64] FLAG: --enable-server="true"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395871    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395875    2576 flags.go:64] FLAG: --event-burst="100"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395879    2576 flags.go:64] FLAG: --event-qps="50"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395882    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395885    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395888    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395893    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395896    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395899    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395902    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395905    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395908    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395911    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395915    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395917    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395920    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395923    2576 flags.go:64] FLAG: --feature-gates=""
Nov 02 21:57:00.400467 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395928    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395931    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395934    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395937    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395942    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395945    2576 flags.go:64] FLAG: --help="false"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395948    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395951    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395954    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395957    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395961    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395964    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395967    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395970    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395974    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395977    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395980    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395983    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395986    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395989    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395992    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395995    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.395998    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 21:57:00.401070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396001    2576 flags.go:64] FLAG: --lock-file=""
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396004    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396007    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396010    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396016    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396019    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396022    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396025    2576 flags.go:64] FLAG: --logging-format="text"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396027    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396031    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396034    2576 flags.go:64] FLAG: --manifest-url=""
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396037    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396041    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396046    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396050    2576 flags.go:64] FLAG: --max-pods="110"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396053    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396056    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396059    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396063    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396066    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396069    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396072    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396080    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396083    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396087    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 21:57:00.402439 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396091    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396093    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396096    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396099    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396102    2576 flags.go:64] FLAG: --port="10250"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396105    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396108    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08690e48e0b48737e"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396111    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396114    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396117    2576 flags.go:64] FLAG: --register-node="true"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396120    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396124    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396127    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396132    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396134    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396138    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396142    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396145    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396148    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396151    2576 flags.go:64] FLAG: --runonce="false"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396155    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396158    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396161    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396164    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396167    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396170    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 21:57:00.403096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396173    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396176    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396179    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396183    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396199    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396211    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396216    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396220    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396223    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396228    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396231    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396234    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396239    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396242    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396245    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396247    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396251    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396254    2576 flags.go:64] FLAG: --v="2"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396259    2576 flags.go:64] FLAG: --version="false"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396264    2576 flags.go:64] FLAG: --vmodule=""
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396268    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396272    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396413    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396416    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396419    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.403771 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396422    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396426    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396429    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396431    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396434    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396437    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396439    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396442    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396445    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396447    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396450    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396452    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396455    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396457    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396460    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396463    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396466    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396468    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396471    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.404392 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396474    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396476    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396478    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396481    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396484    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396486    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396489    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396493    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396496    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396498    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396501    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396504    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396507    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396510    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396514    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396516    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396519    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396522    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396524    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396527    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.404854 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396530    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396532    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396534    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396537    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396540    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396542    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396545    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396547    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396550    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396553    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396556    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396558    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396561    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396563    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396566    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396568    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396571    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396594    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396598    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396602    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.405352 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396606    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396608    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396611    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396613    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396616    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396619    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396624    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396627    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396631    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396635    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396638    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396641    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396644    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396647    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396650    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396653    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396656    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396658    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396661    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.405824 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396663    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396666    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396669    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396671    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396674    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396677    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396679    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396682    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396684    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396687    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396689    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396693    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396698    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396701    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396704    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396706    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396709    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396711    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396714    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.406296 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396718    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396721    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396724    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396726    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396729    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396731    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.396757    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396795    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396800    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396803    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396806    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396809    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396812    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396815    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396817    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396820    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.406756 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396823    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396826    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396828    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396831    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396834    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396836    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396839    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396841    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396844    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396848    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396853    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396857    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396861    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396864    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396867    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396869    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396873    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396876    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.407135 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396878    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396881    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396883    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396886    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396889    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396891    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396894    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396896    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396899    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396902    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396905    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396907    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396910    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396912    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396916    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396920    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396922    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396925    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396928    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.407620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396931    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396934    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396937    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396941    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396944    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396946    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396949    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396951    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396954    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396957    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396960    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396963    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396966    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396969    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396971    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396974    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396977    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396980    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396982    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.408074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396985    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396988    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396990    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396993    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396996    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.396998    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397001    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397003    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397006    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397009    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397011    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397014    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397016    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397019    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397021    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397024    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397027    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397030    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397032    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397035    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.408542 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397037    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397040    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397043    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397045    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397048    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397051    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397054    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397056    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397058    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397061    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397064    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397066    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397069    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397071    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397074    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397077    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397079    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397082    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397084    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.397088    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.409044 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.404314    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.404331    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.404342    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.404346    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404405    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404410    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404413    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404416    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404419    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404422    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404424    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404427    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404430    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404433    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404436    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404438    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404441    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404444    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404447    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404449    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.409531 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404452    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404455    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404457    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404460    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404463    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404465    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404468    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404471    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404474    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404477    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404479    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404482    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404485    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404489    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404491    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404494    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404497    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404499    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404502    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.409990 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404505    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404508    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404510    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404514    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404518    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404521    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404525    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404527    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404530    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404533    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404536    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404538    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404541    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404544    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404547    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404550    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404552    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404555    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404558    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.410483 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404561    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404563    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404566    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404568    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404571    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404574    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404576    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404579    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404584    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404587    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404590    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404593    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404596    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404598    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404601    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404603    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404606    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404609    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404611    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.410945 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404614    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404616    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404619    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404621    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404624    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404626    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404629    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404632    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404636    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404639    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404641    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404646    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404650    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404653    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404656    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404658    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404661    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404664    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.411525 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404667    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404669    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404672    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404675    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404677    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404680    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404682    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404685    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404688    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404690    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404695    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404698    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404701    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.404729    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404889    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.412008 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404893    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404896    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404899    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404901    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404904    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404907    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404909    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404912    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404914    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404918    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404920    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404923    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404928    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404930    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404933    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404936    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404938    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404941    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.412420 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404944    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404947    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404950    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404953    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404955    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404958    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404961    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404963    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404966    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404968    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404971    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404973    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404976    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404978    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404981    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404983    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404986    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404989    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404991    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404994    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.412932 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404996    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.404999    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405002    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405005    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405007    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405010    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405013    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405015    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405018    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405020    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405023    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405025    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405028    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405030    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405033    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405036    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405038    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405041    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405043    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405046    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.413436 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405048    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405051    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405054    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405056    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405059    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405061    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405064    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405066    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405069    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405071    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405074    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405076    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405079    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405083    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405087    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405091    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405094    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405097    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405099    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.413964 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405102    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405104    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405107    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405109    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405112    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405114    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405117    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405119    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405122    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405125    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405127    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405130    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405132    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405135    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405138    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405140    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405143    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405145    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405148    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.414759 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405151    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405155    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405158    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405160    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405163    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405165    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405169    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405171    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.405212    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405246    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405250    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405254    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405257    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405260    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405263    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405266    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:00.415357 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405269    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405272    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405275    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405277    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405280    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405282    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405287    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405290    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405293    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405296    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405299    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405315    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405320    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405323    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405326    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405329    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405332    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405335    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405338    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:00.415739 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405340    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405343    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405346    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405349    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405353    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405356    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405359    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405365    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405367    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405370    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405373    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405376    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405380    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405383    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405385    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405388    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405390    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405393    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:00.416580 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405396    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405399    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405402    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405405    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405407    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405410    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405413    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405415    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405418    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405421    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405423    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405426    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405428    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405431    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405436    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405439    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405442    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405445    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405448    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:00.417066 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405450    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405453    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405456    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405459    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405462    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405464    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405467    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405469    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405472    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405474    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405477    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405480    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405482    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405485    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405489    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405492    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405494    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405497    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405500    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405502    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:00.417604 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405505    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405507    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405510    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405512    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405515    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405518    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405520    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405522    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405525    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405529    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405531    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405534    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405537    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405539    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405542    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405544    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405548    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405550    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405553    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405555    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:00.418099 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:00.405558    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.405612    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.409801    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.410885    2576 server.go:1014] "Starting client certificate rotation"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.410990    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.411036    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.413143    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:57:00.418730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.414216    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:57:00.426459 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.426441    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 02 21:57:00.431242 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.431225    2576 log.go:25] "Validated CRI v1 image API"
Nov 02 21:57:00.432078 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.432053    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 21:57:00.437221 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.437183    2576 fs.go:135] Filesystem UUIDs: map[43cc123c-c5bf-47b5-b122-9ee5f7ab2cac:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7e36c000-ef26-46a5-97b9-d3379b87e499:/dev/nvme0n1p3]
Nov 02 21:57:00.437301 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.437219    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 21:57:00.442606 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.442494    2576 manager.go:217] Machine: {Timestamp:2026-11-02 21:57:00.440967528 +0000 UTC m=+0.378383524 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103914 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d46bdafeb3e8652b609d9602dcebc SystemUUID:ec2d46bd-afeb-3e86-52b6-09d9602dcebc BootID:c4c6a56d-9f8d-4b3d-a016-040a7e464a15 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:61:b0:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:61:b0:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:14:6e:45:cc:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 21:57:00.442606 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.442604    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 21:57:00.442755 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.442738    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 21:57:00.445137 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445120    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 21:57:00.445756 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445729    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 21:57:00.445894 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445757    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-57.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 21:57:00.445943 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445897    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 21:57:00.445943 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445905    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 21:57:00.445997 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445947    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:57:00.445997 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.445965    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:57:00.447328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.447318    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 21:57:00.452040 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452028    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 21:57:00.452152 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452143    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 21:57:00.452208 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452174    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 21:57:00.452208 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452196    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 21:57:00.452295 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452222    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 21:57:00.452295 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452233    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 02 21:57:00.452295 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452242    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 21:57:00.452614 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.452603    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:57:00.455121 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.455105    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 21:57:00.462418 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.462401    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:57:00.463341 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463328    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 21:57:00.463402 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463358    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 21:57:00.463516 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463506    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 21:57:00.463560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463521    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 21:57:00.463560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463528    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 21:57:00.463560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463534    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 21:57:00.463560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463540    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 21:57:00.463560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.463551    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 21:57:00.464257 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464247    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 21:57:00.464257 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464257    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 21:57:00.464322 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464265    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 21:57:00.464322 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464272    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 21:57:00.464322 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464285    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 21:57:00.464322 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464296    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 21:57:00.464686 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.464665    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-57.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 21:57:00.464748 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.464708    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 21:57:00.464909 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464900    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 21:57:00.464943 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.464910    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 21:57:00.468858 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.468844    2576 server.go:1263] "Started kubelet"
Nov 02 21:57:00.468995 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.468944    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 21:57:00.469071 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.469013    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 21:57:00.469071 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.468999    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 21:57:00.469397 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.469382    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v52zp"
Nov 02 21:57:00.469629 ip-10-0-131-57 systemd[1]: Started Kubernetes Kubelet.
Nov 02 21:57:00.474533 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.474419    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-57.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 21:57:00.474792 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.474772    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 21:57:00.477322 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.476882    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 21:57:00.477574 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.477555    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v52zp"
Nov 02 21:57:00.479058 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.479038    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 21:57:00.480571 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.479497    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.us-east-2.compute.internal.18e4593170124a6e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 21:57:00.468816494 +0000 UTC m=+0.406232489,LastTimestamp:2026-11-02 21:57:00.468816494 +0000 UTC m=+0.406232489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.us-east-2.compute.internal,}"
Nov 02 21:57:00.484981 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.484958    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 21:57:00.485070 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.484979    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 21:57:00.485070 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.485053    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.485709 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.485678    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 21:57:00.485896 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.485879    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 21:57:00.485896 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.485894    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 21:57:00.486032 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.485976    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 21:57:00.486032 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.485986    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 21:57:00.486302 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486286    2576 factory.go:55] Registering systemd factory
Nov 02 21:57:00.486390 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486304    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 02 21:57:00.486508 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486497    2576 factory.go:153] Registering CRI-O factory
Nov 02 21:57:00.486573 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486510    2576 factory.go:223] Registration of the crio container factory successfully
Nov 02 21:57:00.486573 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486562    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 21:57:00.486668 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486582    2576 factory.go:103] Registering Raw factory
Nov 02 21:57:00.486668 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.486597    2576 manager.go:1199] Started watching for new ooms in manager
Nov 02 21:57:00.487967 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.487952    2576 manager.go:319] Starting recovery of all containers
Nov 02 21:57:00.496509 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.496481    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:00.497045 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.497019    2576 manager.go:324] Recovery completed
Nov 02 21:57:00.499289 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.499268    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-57.us-east-2.compute.internal\" not found" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.501523 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.501510    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.503199 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503170    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.503284 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503222    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.503284 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503234    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.503785 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503771    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 02 21:57:00.503785 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503783    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.503930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503782    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 21:57:00.503930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.503814    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 21:57:00.504480 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.504464    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.504580 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.504497    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.504580 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.504512    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.505933 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.505922    2576 policy_none.go:50] "Start"
Nov 02 21:57:00.505967 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.505937    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 21:57:00.505967 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.505946    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 21:57:00.507928 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.507918    2576 policy_none.go:44] "Start"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.556864    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.558014    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.558046    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.558077    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.558123    2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 21:57:00.564138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.561307    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:00.585812 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.585780    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.658432 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.658404    2576 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 21:57:00.678820 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.678785    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 21:57:00.681178 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681165    2576 manager.go:341] "Starting Device Plugin manager"
Nov 02 21:57:00.681268 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.681214    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 21:57:00.681268 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681226    2576 server.go:87] "Starting device plugin registration server"
Nov 02 21:57:00.681595 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681580    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 21:57:00.681629 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681600    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 21:57:00.681783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681708    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 21:57:00.681856 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681816    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 21:57:00.681856 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.681825    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 21:57:00.682344 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.682323    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 21:57:00.682443 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.682377    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.782713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.782679    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.783794 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.783778    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.783874 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.783810    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.783874 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.783820    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.783874 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.783853    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.790785 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.790767    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.790828 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.790793    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-57.us-east-2.compute.internal\": node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.803533 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.803508    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.859244 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.859179    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"]
Nov 02 21:57:00.859373 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.859285    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.860212 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.860171    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.860320 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.860227    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.860320 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.860243    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.862561 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.862550    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.862755 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.862739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.862807 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.862769    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.863241 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863226    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.863318 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863250    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.863318 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863269    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.863318 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863281    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.863454 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863255    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.863454 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.863348    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.865512 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.865494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.865593 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.865526    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:00.866249 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.866232    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:00.866352 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.866269    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:00.866352 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.866286    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:00.887787 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.887764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.887901 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.887795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.887901 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.887863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.890717 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.890679    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.892652 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.892637    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.903988 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:00.903971    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:00.988539 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.988539 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.988759 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.988759 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.988759 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988638    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:00.988759 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:00.988632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:01.004654 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:01.004625    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:01.105125 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:01.105083    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:02.712927 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:02.712900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:02.715016 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:02.714987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:02.722556 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:02.722529    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:02.823136 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:02.822950    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:02.923434 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:02.923402    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:02.927578 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:02.927559    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 21:57:02.927725 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:02.927705    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 21:57:02.927802 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:02.927725    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 21:57:03.001687 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.001606    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:52:01 +0000 UTC" deadline="2028-05-18 15:36:25.862021089 +0000 UTC"
Nov 02 21:57:03.001687 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.001650    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13505h39m22.860387754s"
Nov 02 21:57:03.023663 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.023635    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:03.124174 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.124138    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:03.224789 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.224757    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:03.325194 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.325121    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:03.328203 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.328184    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:03.379861 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:03.379819    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664740e8a5efdbcb60ae50b0e894ce2d.slice/crio-ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817 WatchSource:0}: Error finding container ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817: Status 404 returned error can't find the container with id ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817
Nov 02 21:57:03.380057 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:03.380037    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd522352ece329563a2383a55c8031694.slice/crio-ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f WatchSource:0}: Error finding container ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f: Status 404 returned error can't find the container with id ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f
Nov 02 21:57:03.385256 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.385241    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:57:03.411066 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.411045    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:03.503595 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.503549    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:03.514570 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.514530    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:03.515697 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.515685    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:03.524169 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.524154    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:03.811793 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.811672    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:03.969960 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.969919    2576 apiserver.go:52] "Watching apiserver"
Nov 02 21:57:03.977887 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.977825    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 21:57:03.978201 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.978182    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv","openshift-dns/node-resolver-xcrch","openshift-network-operator/iptables-alerter-6rjz9","kube-system/global-pull-secret-syncer-bvq2f","openshift-image-registry/node-ca-85npx","openshift-multus/network-metrics-daemon-nvvxb","openshift-network-diagnostics/network-check-target-kbg45","openshift-multus/multus-additional-cni-plugins-vxz77","openshift-multus/multus-hclsq","openshift-ovn-kubernetes/ovnkube-node-55qz7","openshift-cluster-node-tuning-operator/tuned-r9brh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal","kube-system/konnectivity-agent-kzxpt"]
Nov 02 21:57:03.987168 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.987137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:03.987285 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.987240    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:03.989594 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.989571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:03.991852 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.991831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:03.993199 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.993181    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 21:57:03.993494 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.993475    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtxk8\""
Nov 02 21:57:03.993919 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.993902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:03.994255 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.994240    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 21:57:03.994401 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.994385    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 21:57:03.994508 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.994495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 21:57:03.994996 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.994978    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 21:57:03.995168 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.995125    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 21:57:03.995306 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.995191    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 21:57:03.996398 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.996378    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n7kkn\""
Nov 02 21:57:03.996503 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.996469    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 21:57:03.996610 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.996577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:03.996670 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:03.996638    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:03.996722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.996675    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf8xp\""
Nov 02 21:57:03.997978 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.997956    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:03.998235 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.998142    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:03.998235 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.998162    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 21:57:03.998838 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:03.998817    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.000993 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.000971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.001936 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.001914    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 21:57:04.002708 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.002686    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 21:57:04.003794 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.003776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.005804 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.005786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.008177 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.008155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.011477 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.011461    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:04.011600 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.011515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:04.013391 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.013486 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.013486 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.013614 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.013614 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.013697 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxsr\" (UniqueName: \"kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.013697 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.013782 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.013998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l427q\" (UniqueName: \"kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7mh\" (UniqueName: \"kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.014510 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grxx\" (UniqueName: \"kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt568\" (UniqueName: \"kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015144 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.014990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpsh\" (UniqueName: \"kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.015861 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.015455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.033857 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.033815    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 21:57:04.033983 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.033888    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g94rr"
Nov 02 21:57:04.039930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.039879    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g94rr"
Nov 02 21:57:04.081552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.081452    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"d522352ece329563a2383a55c8031694","Type":"ContainerStarted","Data":"ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f"}
Nov 02 21:57:04.083201 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.083175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerStarted","Data":"ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817"}
Nov 02 21:57:04.094395 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.094369    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:04.094527 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.094505    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 21:57:04.094619 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.094607    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 21:57:04.094675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.094637    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 21:57:04.094976 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.094961    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 21:57:04.095031 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095016    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 21:57:04.095182 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095168    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 21:57:04.095516 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095499    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tk4z\""
Nov 02 21:57:04.095596 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095589    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 21:57:04.095651 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095645    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sd22g\""
Nov 02 21:57:04.095701 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095679    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 21:57:04.095749 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095719    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dx8bh\""
Nov 02 21:57:04.095749 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.095747    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 21:57:04.096176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096161    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 21:57:04.096354 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096341    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 21:57:04.096435 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096422    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7q497\""
Nov 02 21:57:04.096489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096482    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 21:57:04.096563 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096551    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-447sr\""
Nov 02 21:57:04.096620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096608    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 21:57:04.096669 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096640    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:04.096721 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.096679    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4qcf6\""
Nov 02 21:57:04.104415 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.104395    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 21:57:04.115863 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.115838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.115971 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.115869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.115971 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.115886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.115971 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.115909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.116086 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.115970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116086 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116086 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.116192 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.116192 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116192 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.116328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.116328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.116328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.116461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.116461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7mh\" (UniqueName: \"kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.116675 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt568\" (UniqueName: \"kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117029 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.116989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.117210    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:04.117572 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.117297    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:04.617276397 +0000 UTC m=+3.037346822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:04.117793 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.117793 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117793 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.117908 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.117908 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117874    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.117908 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.117982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.118003    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:04.118015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxsr\" (UniqueName: \"kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118261 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.118036    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:04.6180252 +0000 UTC m=+3.038095610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:04.118261 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118261 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.118494 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118617 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.118617 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.118742 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.118742 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.118846 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjs5\" (UniqueName: \"kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.118846 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.118946 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.118946 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.119050 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.118951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.119050 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l427q\" (UniqueName: \"kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.119147 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.119147 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.119147 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.119291 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.119620 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119682    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119707    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grxx\" (UniqueName: \"kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.119945 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.119983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120076    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120114    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xbp\" (UniqueName: \"kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.120337 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.121041 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpsh\" (UniqueName: \"kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.121041 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.121041 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.121041 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.121041 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.120626    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 21:57:04.121278 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.121064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.122096 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.121894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.122213 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.122191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.124920 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.124903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.125012 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.124998    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.134464 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.134424    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7mh\" (UniqueName: \"kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.135453 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.135423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt568\" (UniqueName: \"kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.136034 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.136010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l427q\" (UniqueName: \"kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.136209 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.136186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxsr\" (UniqueName: \"kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.136919 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.136895    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grxx\" (UniqueName: \"kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.137760 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.137739    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpsh\" (UniqueName: \"kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.139692 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.139671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.221261 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjs5\" (UniqueName: \"kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221483    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.221750 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xbp\" (UniqueName: \"kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221912    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.221988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222142    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.222379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222246    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222288    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222451    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222478    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222481    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.223327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.222576    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.224737 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.224709    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.224891 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.224868    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.226906 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.226886    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:04.227000 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.226910    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:04.227000 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.226925    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:04.227000 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.226985    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:04.726967982 +0000 UTC m=+3.147038415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:04.230495 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.230468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xbp\" (UniqueName: \"kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.230920 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.230895    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjs5\" (UniqueName: \"kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.277405 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.277372    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:04.320730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.320652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:04.322911 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.322886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:04.330147 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.330128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:04.332590 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.332566    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf18c217_668e_4f22_8bb3_ea403e33e962.slice/crio-299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0 WatchSource:0}: Error finding container 299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0: Status 404 returned error can't find the container with id 299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0
Nov 02 21:57:04.333227 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.333206    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f5aae5_63bc_4e64_a357_1b046c4fc470.slice/crio-0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a WatchSource:0}: Error finding container 0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a: Status 404 returned error can't find the container with id 0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a
Nov 02 21:57:04.333372 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.333355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:04.342490 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.342457    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2571db_e8bb_431c_a74d_dea6f9a261cc.slice/crio-278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2 WatchSource:0}: Error finding container 278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2: Status 404 returned error can't find the container with id 278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2
Nov 02 21:57:04.343821 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.343773    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afa6f34_35a6_4bcd_939e_8083b7cb4fb3.slice/crio-05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99 WatchSource:0}: Error finding container 05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99: Status 404 returned error can't find the container with id 05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99
Nov 02 21:57:04.363111 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.363086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:04.365208 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.365182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:04.372030 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.372013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:04.372368 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.372253    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f05bd7_b4eb_4df9_af57_fb0277839916.slice/crio-cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7 WatchSource:0}: Error finding container cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7: Status 404 returned error can't find the container with id cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7
Nov 02 21:57:04.374871 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.374849    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36174814_afd3_42b6_a555_75c832ebacf5.slice/crio-6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b WatchSource:0}: Error finding container 6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b: Status 404 returned error can't find the container with id 6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b
Nov 02 21:57:04.375559 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.375521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hclsq"
Nov 02 21:57:04.382653 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.382099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:04.383605 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.383477    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d91b2f9_d641_449a_9bae_601f629cc281.slice/crio-09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6 WatchSource:0}: Error finding container 09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6: Status 404 returned error can't find the container with id 09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6
Nov 02 21:57:04.385074 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.385042    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a19a824_6d01_4257_a6c5_90c2d7f0c649.slice/crio-9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86 WatchSource:0}: Error finding container 9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86: Status 404 returned error can't find the container with id 9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86
Nov 02 21:57:04.392049 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:04.392015    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f55ac_11b3_442a_beb6_f2aadf8aa899.slice/crio-ce549dbd4f2481f935200e7cb1f83b295d0f55011e1d7848ec099fafa5585e18 WatchSource:0}: Error finding container ce549dbd4f2481f935200e7cb1f83b295d0f55011e1d7848ec099fafa5585e18: Status 404 returned error can't find the container with id ce549dbd4f2481f935200e7cb1f83b295d0f55011e1d7848ec099fafa5585e18
Nov 02 21:57:04.625704 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.625630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:04.625704 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.625671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:04.625915 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.625789    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:04.625915 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.625846    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:05.625834144 +0000 UTC m=+4.045904555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:04.625915 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.625789    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:04.626114 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.625937    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:05.625924104 +0000 UTC m=+4.045994532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:04.827449 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:04.827413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:04.827666 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.827647    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:04.827783 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.827672    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:04.827783 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.827684    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:04.827783 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:04.827749    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:05.827731303 +0000 UTC m=+4.247801730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:05.043877 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.043782    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:04 +0000 UTC" deadline="2028-07-22 14:09:30.643914166 +0000 UTC"
Nov 02 21:57:05.043877 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.043834    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h12m25.600082268s"
Nov 02 21:57:05.109963 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.109919    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv" event={"ID":"8d91b2f9-d641-449a-9bae-601f629cc281","Type":"ContainerStarted","Data":"09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6"}
Nov 02 21:57:05.117183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.117087    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kzxpt" event={"ID":"36174814-afd3-42b6-a555-75c832ebacf5","Type":"ContainerStarted","Data":"6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b"}
Nov 02 21:57:05.122336 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.122235    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerStarted","Data":"cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7"}
Nov 02 21:57:05.129251 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.129205    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85npx" event={"ID":"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3","Type":"ContainerStarted","Data":"05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99"}
Nov 02 21:57:05.141497 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.141463    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a"}
Nov 02 21:57:05.146377 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.146347    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9brh" event={"ID":"342f55ac-11b3-442a-beb6-f2aadf8aa899","Type":"ContainerStarted","Data":"ce549dbd4f2481f935200e7cb1f83b295d0f55011e1d7848ec099fafa5585e18"}
Nov 02 21:57:05.171799 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.171754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6rjz9" event={"ID":"0a2571db-e8bb-431c-a74d-dea6f9a261cc","Type":"ContainerStarted","Data":"278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2"}
Nov 02 21:57:05.181490 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.181453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcrch" event={"ID":"bf18c217-668e-4f22-8bb3-ea403e33e962","Type":"ContainerStarted","Data":"299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0"}
Nov 02 21:57:05.183270 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.183199    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hclsq" event={"ID":"6a19a824-6d01-4257-a6c5-90c2d7f0c649","Type":"ContainerStarted","Data":"9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86"}
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.633876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.633994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.634016    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.634085    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:07.634068235 +0000 UTC m=+6.054138661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.634109    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:05.634178 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.634145    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:07.63413372 +0000 UTC m=+6.054204143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:05.836623 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:05.836588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:05.836815 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.836798    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:05.836899 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.836823    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:05.836899 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.836835    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:05.836899 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:05.836890    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:07.836872552 +0000 UTC m=+6.256942969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:06.045025 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.044973    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:04 +0000 UTC" deadline="2028-08-17 23:40:21.401508476 +0000 UTC"
Nov 02 21:57:06.045025 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.045003    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15697h43m15.356508602s"
Nov 02 21:57:06.083691 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.083664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:06.083844 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:06.083771    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:06.084104 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.084086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:06.084175 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:06.084165    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:06.084327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.084312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:06.084410 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:06.084392    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:06.202138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.201495    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"d522352ece329563a2383a55c8031694","Type":"ContainerStarted","Data":"e63262c6d389993c19f572d542ab27a4249d5788e75ce020e25e7463ee779388"}
Nov 02 21:57:06.211605 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.211414    2576 generic.go:356] "Generic (PLEG): container finished" podID="664740e8a5efdbcb60ae50b0e894ce2d" containerID="01709f200936b9e8f85ae1390c31c3ff25a48afba1c3ccb138a6f664fdc8d27e" exitCode=0
Nov 02 21:57:06.211605 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.211453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerDied","Data":"01709f200936b9e8f85ae1390c31c3ff25a48afba1c3ccb138a6f664fdc8d27e"}
Nov 02 21:57:06.217652 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:06.217585    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal" podStartSLOduration=3.217567929 podStartE2EDuration="3.217567929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:06.216478734 +0000 UTC m=+4.636549159" watchObservedRunningTime="2026-11-02 21:57:06.217567929 +0000 UTC m=+4.637638364"
Nov 02 21:57:07.226743 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:07.226033    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerStarted","Data":"3041ac3e94357d75e0b7b9dda4d4a6a8f7c5b3f2f3eccd7388e31567a1387b81"}
Nov 02 21:57:07.236589 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:07.236514    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerStarted","Data":"1c2aeb157ffe4e5dee870b912471b88744a394ca71b13e5874b2988c682ae4ad"}
Nov 02 21:57:07.649638 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:07.649596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:07.649829 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:07.649656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:07.649897 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.649830    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:07.649897 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.649889    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:11.649872219 +0000 UTC m=+10.069942650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:07.650008 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.649957    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:07.650008 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.649992    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:11.649979706 +0000 UTC m=+10.070050131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:07.852137 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:07.851490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:07.852137 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.851687    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:07.852137 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.851711    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:07.852137 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.851723    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:07.852137 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:07.851791    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:11.85177081 +0000 UTC m=+10.271841244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:08.076744 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.076662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:08.076900 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:08.076808    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:08.077211 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.077179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:08.077345 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:08.077269    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:08.079349 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.079125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:08.079349 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:08.079239    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:08.249074 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.247924    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3041ac3e94357d75e0b7b9dda4d4a6a8f7c5b3f2f3eccd7388e31567a1387b81" exitCode=0
Nov 02 21:57:08.249074 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.248860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"3041ac3e94357d75e0b7b9dda4d4a6a8f7c5b3f2f3eccd7388e31567a1387b81"}
Nov 02 21:57:08.270552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:08.269334    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" podStartSLOduration=5.269317726 podStartE2EDuration="5.269317726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:07.265257291 +0000 UTC m=+5.685327728" watchObservedRunningTime="2026-11-02 21:57:08.269317726 +0000 UTC m=+6.689388160"
Nov 02 21:57:09.257103 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:09.257020    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kzxpt" event={"ID":"36174814-afd3-42b6-a555-75c832ebacf5","Type":"ContainerStarted","Data":"1c59b51a86d0a1c4900e2d863dbd52265239e994291c39d709fcc753db987a5a"}
Nov 02 21:57:09.274237 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:09.269203    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85npx" event={"ID":"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3","Type":"ContainerStarted","Data":"a7eddfeba6d21a7ef17b692444b6dc98e2570f42d2b23b585ef0915eca8268fc"}
Nov 02 21:57:09.296381 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:09.295800    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kzxpt" podStartSLOduration=3.531214513 podStartE2EDuration="7.295781765s" totalImagesPullingTime="3.764567252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:09.282162695 +0000 UTC m=+7.702233129" watchObservedRunningTime="2026-11-02 21:57:09.295781765 +0000 UTC m=+7.715852199"
Nov 02 21:57:09.296381 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:09.296138    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-85npx" podStartSLOduration=2.655431491 podStartE2EDuration="7.296129804s" totalImagesPullingTime="4.640698313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:09.295146363 +0000 UTC m=+7.715216798" watchObservedRunningTime="2026-11-02 21:57:09.296129804 +0000 UTC m=+7.716200240"
Nov 02 21:57:10.079552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:10.078749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:10.079552 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:10.078889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:10.079552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:10.079301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:10.079552 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:10.079391    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:10.079552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:10.079448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:10.080022 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:10.079520    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:11.277930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:11.277849    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv" event={"ID":"8d91b2f9-d641-449a-9bae-601f629cc281","Type":"ContainerStarted","Data":"2f8bd39ac87306a958d3ac8cc12bc63e46c77dbec80f0de4740b7afc983a51c0"}
Nov 02 21:57:11.685000 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:11.684960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:11.685199 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:11.685019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:11.685268 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.685235    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:11.685316 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.685297    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:19.68527915 +0000 UTC m=+18.105349566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:11.685792 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.685767    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:11.685905 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.685820    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:19.685805966 +0000 UTC m=+18.105876382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:11.886762 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:11.886727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:11.886959 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.886915    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:11.886959 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.886937    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:11.886959 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.886949    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:11.887125 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:11.887001    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:19.886986745 +0000 UTC m=+18.307057157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:12.076813 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.076789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:12.076938 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:12.076915    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:12.076988 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.076975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:12.077070 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:12.077054    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:12.077922 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.077417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:12.077922 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:12.077515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:12.104012 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.103991    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 21:57:12.207486 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.206606    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:57:12.104009089Z","UUID":"ee929ef6-42fa-48c7-9bc1-ca588ddce0fd","Handler":null,"Name":"","Endpoint":""}
Nov 02 21:57:12.208709 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.208213    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 21:57:12.208709 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.208247    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 21:57:12.284328 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.283552    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcrch" event={"ID":"bf18c217-668e-4f22-8bb3-ea403e33e962","Type":"ContainerStarted","Data":"251ed47f9b6d3e33bf145b6804df23e1861ad26a7a7c83da01f843f4e90d4eab"}
Nov 02 21:57:12.286848 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.286333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hclsq" event={"ID":"6a19a824-6d01-4257-a6c5-90c2d7f0c649","Type":"ContainerStarted","Data":"932844558d6406284acb22a28c42e3d96883f2afed9d09002d32650ec956ab25"}
Nov 02 21:57:12.289813 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.288648    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv" event={"ID":"8d91b2f9-d641-449a-9bae-601f629cc281","Type":"ContainerStarted","Data":"be7eaf381a85b731734bb38660ec3fa193158855f6aba2549fae85ed516efac8"}
Nov 02 21:57:12.321273 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.321191    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xcrch" podStartSLOduration=2.600449598 podStartE2EDuration="10.321175249s" totalImagesPullingTime="7.720725651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:12.301427555 +0000 UTC m=+10.721497988" watchObservedRunningTime="2026-11-02 21:57:12.321175249 +0000 UTC m=+10.741245681"
Nov 02 21:57:12.659975 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.659732    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:12.660712 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.660616    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:12.682028 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:12.681977    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hclsq" podStartSLOduration=3.081040925 podStartE2EDuration="10.681959339s" totalImagesPullingTime="7.600918414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:12.321409329 +0000 UTC m=+10.741479761" watchObservedRunningTime="2026-11-02 21:57:12.681959339 +0000 UTC m=+11.102029773"
Nov 02 21:57:13.294359 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:13.294325    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3a6ab2e430674a2d393360d5f65d20c4d9f3273bc055c0a78da0168d13df1664" exitCode=0
Nov 02 21:57:13.294782 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:13.294473    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"3a6ab2e430674a2d393360d5f65d20c4d9f3273bc055c0a78da0168d13df1664"}
Nov 02 21:57:14.076782 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.076561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:14.076782 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.076778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:14.076949 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:14.076866    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:14.076949 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.076942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:14.077029 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:14.077019    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:14.077097 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:14.077074    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:14.298955 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.298868    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv" event={"ID":"8d91b2f9-d641-449a-9bae-601f629cc281","Type":"ContainerStarted","Data":"5d8515d3311c59084fcf4935c85cf9a45e748769bc5ddf88752b1a3e7cd6a16b"}
Nov 02 21:57:14.301360 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.301333    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="bffe358a47a34b9a5e0c810a657f66e2cb4872d0c5f43a68da0c1500f62fac65" exitCode=0
Nov 02 21:57:14.301495 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.301409    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"bffe358a47a34b9a5e0c810a657f66e2cb4872d0c5f43a68da0c1500f62fac65"}
Nov 02 21:57:14.303856 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.303829    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6rjz9" event={"ID":"0a2571db-e8bb-431c-a74d-dea6f9a261cc","Type":"ContainerStarted","Data":"bf9cf0216adf3ee700a765c0ed40a092266a2165d21f16073fd2a6d0d3ae7db6"}
Nov 02 21:57:14.316315 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.316265    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv" podStartSLOduration=3.949432341 podStartE2EDuration="12.316252824s" totalImagesPullingTime="8.366820483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:14.31512837 +0000 UTC m=+12.735198804" watchObservedRunningTime="2026-11-02 21:57:14.316252824 +0000 UTC m=+12.736323252"
Nov 02 21:57:14.346057 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:14.346010    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6rjz9" podStartSLOduration=4.286175398 podStartE2EDuration="12.34599615s" totalImagesPullingTime="8.059820752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:14.328174724 +0000 UTC m=+12.748245156" watchObservedRunningTime="2026-11-02 21:57:14.34599615 +0000 UTC m=+12.766066583"
Nov 02 21:57:15.309522 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:15.309433    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3f0697db4bccbe41758964f4626ac82e5d4ef1dc3e8ffc173bb624c0c5da457a" exitCode=0
Nov 02 21:57:15.309522 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:15.309497    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"3f0697db4bccbe41758964f4626ac82e5d4ef1dc3e8ffc173bb624c0c5da457a"}
Nov 02 21:57:15.311453 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:15.311395    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9brh" event={"ID":"342f55ac-11b3-442a-beb6-f2aadf8aa899","Type":"ContainerStarted","Data":"3421e0f9ff0855a22d19714ddfc05b2a0849a3c263c35868827cba8cc4e2b885"}
Nov 02 21:57:15.344034 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:15.343987    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r9brh" podStartSLOduration=3.084738237 podStartE2EDuration="13.343970383s" totalImagesPullingTime="10.259232146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:15.343434703 +0000 UTC m=+13.763505137" watchObservedRunningTime="2026-11-02 21:57:15.343970383 +0000 UTC m=+13.764040823"
Nov 02 21:57:16.076332 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:16.076297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:16.076508 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:16.076330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:16.076508 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:16.076416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:16.076508 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:16.076479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:16.076691 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:16.076589    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:16.076691 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:16.076658    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:17.319398 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:17.319357    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerStarted","Data":"85cbe655f6580206745845c32f5c48e4472b9efab22bdf0b07f84060d4f792b1"}
Nov 02 21:57:18.076203 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:18.076164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:18.076388 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:18.076170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:18.076388 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:18.076318    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:18.076388 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:18.076170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:18.076503 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:18.076386    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:18.076503 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:18.076462    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:18.324089 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:18.324053    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="85cbe655f6580206745845c32f5c48e4472b9efab22bdf0b07f84060d4f792b1" exitCode=0
Nov 02 21:57:18.324434 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:18.324100    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"85cbe655f6580206745845c32f5c48e4472b9efab22bdf0b07f84060d4f792b1"}
Nov 02 21:57:19.329618 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329395    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"642a713d83700a33c86c053ce2166312b4d5530809fe191b79dc180eb127dfe2"}
Nov 02 21:57:19.330176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329705    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"44b939c93153c8a31ac716fc2b8e9e95d48097f1a34a02d63625a56ee74d360d"}
Nov 02 21:57:19.330176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329762    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"02406808711d07f13868a18e3d3b30cfeb4587333418e7706e69c8165006ec57"}
Nov 02 21:57:19.330176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329814    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"501ca10e4f48232ede7f7053e54adf5bfbd27f7ce9a855e9472f4b919d402cfe"}
Nov 02 21:57:19.330176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329856    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"ca8a7f3531d7cf2c4ac701e6b07b910ef3c6f8ea2679f0807185ef0b11ba317d"}
Nov 02 21:57:19.330176 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.329897    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"0a278e201d75cdd417126ab246683790057a49530dce5af82ca4a935e7c20fa8"}
Nov 02 21:57:19.332284 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.332259    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="5c9929ac0e73521660521a8bec2ae7d519b0c1f9897f31a135651d45e12c7082" exitCode=0
Nov 02 21:57:19.332393 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.332291    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerDied","Data":"5c9929ac0e73521660521a8bec2ae7d519b0c1f9897f31a135651d45e12c7082"}
Nov 02 21:57:19.739036 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.738805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:19.739036 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.739005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:19.739235 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.738947    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:19.739235 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.739102    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.739088723 +0000 UTC m=+34.159159149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:57:19.739235 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.739115    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:19.739235 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.739163    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret podName:9d39c020-6a26-47a2-8f57-1655acdc05f0 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.739147049 +0000 UTC m=+34.159217475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret") pod "global-pull-secret-syncer-bvq2f" (UID: "9d39c020-6a26-47a2-8f57-1655acdc05f0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:57:19.940097 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:19.940065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:19.940239 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.940203    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:57:19.940239 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.940218    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:57:19.940239 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.940227    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pn7wv for pod openshift-network-diagnostics/network-check-target-kbg45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:19.940338 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:19.940276    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv podName:8031ebee-f5f9-4efa-acac-f56c132e93cf nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.940262937 +0000 UTC m=+34.360333363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pn7wv" (UniqueName: "kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv") pod "network-check-target-kbg45" (UID: "8031ebee-f5f9-4efa-acac-f56c132e93cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:57:20.076549 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:20.076445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:20.076549 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:20.076464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:20.076755 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:20.076554    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:20.076755 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:20.076653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:20.076856 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:20.076761    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:20.076856 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:20.076825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:20.337441 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:20.337357    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vxz77" event={"ID":"b2f05bd7-b4eb-4df9-af57-fb0277839916","Type":"ContainerStarted","Data":"682509e4841cd74c632a88c0d8cb28394815d9ad46211d1489beb552135e4cef"}
Nov 02 21:57:20.359658 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:20.359609    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vxz77" podStartSLOduration=8.131506552 podStartE2EDuration="18.359597303s" totalImagesPullingTime="10.228090751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:20.358067128 +0000 UTC m=+18.778137543" watchObservedRunningTime="2026-11-02 21:57:20.359597303 +0000 UTC m=+18.779667735"
Nov 02 21:57:21.343089 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:21.343055    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"fe17e0743935a1dfe33ea05fab05671d3febfdcff1e7139df530d0b5d164189b"}
Nov 02 21:57:22.076812 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.076774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:22.077001 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.076900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:22.077001 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:22.076913    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:22.077001 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:22.076985    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:22.077001 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.076998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:22.077147 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:22.077079    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:22.159762 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.159735    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:22.159918 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.159845    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:22.160311 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:22.160295    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:23.351972 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:23.351935    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" event={"ID":"e0f5aae5-63bc-4e64-a357-1b046c4fc470","Type":"ContainerStarted","Data":"ad9ee1a175baf359b59ec7a4fd64a264e2acaaa0d7b3fb4513173bbd9ba904ca"}
Nov 02 21:57:23.352522 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:23.352240    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:23.366919 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:23.366889    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:23.378624 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:23.378577    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7" podStartSLOduration=6.878172568 podStartE2EDuration="21.378563482s" totalImagesPullingTime="14.500390914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:23.376487437 +0000 UTC m=+21.796557871" watchObservedRunningTime="2026-11-02 21:57:23.378563482 +0000 UTC m=+21.798633914"
Nov 02 21:57:24.075726 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.075683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:24.075726 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.075701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:24.075930 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:24.075853    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:24.075930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.075896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:24.076007 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:24.075977    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:24.076044 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:24.076034    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:24.353992 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.353970    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:24.354496 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.354407    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:24.369231 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:24.369211    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:25.015008 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.014976    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvq2f"]
Nov 02 21:57:25.015166 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.015152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:25.015296 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:25.015270    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:25.018637 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.018604    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbg45"]
Nov 02 21:57:25.018778 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.018710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:25.018846 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:25.018821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:25.019486 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.019464    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nvvxb"]
Nov 02 21:57:25.019598 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.019584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:25.019691 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:25.019675    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:25.356249 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:25.356222    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:26.358870 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:26.358841    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:27.076149 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:27.075966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:27.076297 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:27.076021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:27.076297 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:27.076221    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:27.076382 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:27.076021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:27.076382 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:27.076305    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:27.076436 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:27.076394    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:27.230683 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:27.230657    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:29.076549 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:29.076503    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:29.076549 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:29.076549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:29.076947 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:29.076665    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:29.076947 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:29.076704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:29.076947 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:29.076773    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:29.076947 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:29.076839    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:31.075776 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.075744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:31.075776 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.075772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:31.076179 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.075744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:31.076179 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:31.075850    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvvxb" podUID="b911671e-958f-441e-92cc-bf023d0757a7"
Nov 02 21:57:31.076179 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:31.075912    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvq2f" podUID="9d39c020-6a26-47a2-8f57-1655acdc05f0"
Nov 02 21:57:31.076179 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:31.075978    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbg45" podUID="8031ebee-f5f9-4efa-acac-f56c132e93cf"
Nov 02 21:57:31.896198 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.896173    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeReady"
Nov 02 21:57:31.896353 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.896298    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 21:57:31.927864 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.927824    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plm4n"]
Nov 02 21:57:31.932055 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.932028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:31.932566 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.932449    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x8bf8"]
Nov 02 21:57:31.935937 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.935910    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q9m4v"]
Nov 02 21:57:31.936057 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.936040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:31.938697 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.938679    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jx4w4"]
Nov 02 21:57:31.938799 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.938785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:31.941349 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.941333    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"]
Nov 02 21:57:31.941455 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.941431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:31.944240 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.944205    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 21:57:31.944500 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.944303    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 21:57:31.944705 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.944689    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"]
Nov 02 21:57:31.945011 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.944950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:31.948081 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.948005    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 21:57:31.948506 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.948482    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"]
Nov 02 21:57:31.948730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.948709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:31.955489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.955469    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:57:31.955682 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.955635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:31.960693 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.959597    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"]
Nov 02 21:57:31.960693 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.959939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:31.962813 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.962793    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"]
Nov 02 21:57:31.963616 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.963588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:31.973605 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.972203    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"]
Nov 02 21:57:31.973605 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.972365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:31.978919 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.978897    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8469b8f4f5-bf9r8"]
Nov 02 21:57:31.979148 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.979123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:31.982342 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.982321    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:57:31.982583 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.982450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:31.985273 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.985251    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"]
Nov 02 21:57:31.985387 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.985370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:31.988776 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.988758    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"]
Nov 02 21:57:31.988898 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.988880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:31.991781 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991760    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plm4n"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991792    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x8bf8"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991808    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q9m4v"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991830    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jx4w4"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991844    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991857    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"]
Nov 02 21:57:31.991876 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991875    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"]
Nov 02 21:57:31.992431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991900    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8469b8f4f5-bf9r8"]
Nov 02 21:57:31.992431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991922    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kcsw9"]
Nov 02 21:57:31.992431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.991965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:31.995120 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.995101    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lknc7"]
Nov 02 21:57:31.995193 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.995181    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:31.999702 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999677    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:57:31.999783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999714    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"]
Nov 02 21:57:31.999783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999744    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"]
Nov 02 21:57:31.999783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999770    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"]
Nov 02 21:57:31.999783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999784    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"]
Nov 02 21:57:31.999936 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999796    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"]
Nov 02 21:57:31.999936 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:31.999936 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999813    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kcsw9"]
Nov 02 21:57:32.000042 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:31.999983    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lknc7"]
Nov 02 21:57:32.000042 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.000013    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:57:32.024198 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.024287 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.024287 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.024354 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.024428 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.024461 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.024490 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhs5p\" (UniqueName: \"kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.024490 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.024559 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhv2\" (UniqueName: \"kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.024614 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.024655 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.024655 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.024717 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5br8\" (UniqueName: \"kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8\") pod \"kas-connection-checker-685895c7c9-plm4n\" (UID: \"b103cd80-a503-475f-b9a8-f448fbd0af73\") " pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:32.024746 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.024777 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.024814 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.024846 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb25\" (UniqueName: \"kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.024915 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.024944 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhx6\" (UniqueName: \"kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.024974 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.024940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bp6\" (UniqueName: \"kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.025015 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.025046 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.025079 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.025119 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q5w6\" (UniqueName: \"kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6\") pod \"volume-data-source-validator-58c9f5f46d-grv5x\" (UID: \"fbd984d3-01fe-4b43-9ac1-4259e6c39faa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:32.025148 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.025148 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.025205 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.025205 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.025263 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qkb\" (UniqueName: \"kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb\") pod \"kas-connection-checker-685895c7c9-q9m4v\" (UID: \"0530c38c-3055-48b0-a829-ad3068c97528\") " pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:32.025301 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr94q\" (UniqueName: \"kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q\") pod \"kas-connection-checker-685895c7c9-x8bf8\" (UID: \"3a2a4927-992c-4f11-9f42-81405452f431\") " pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:32.025383 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.025365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.050465 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050432    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 21:57:32.050570 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050498    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 21:57:32.050570 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050502    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.050869 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050841    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 21:57:32.050946 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050879    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.050946 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050913    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.050946 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050940    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 21:57:32.051093 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.050949    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.051093 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051017    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 21:57:32.051093 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051076    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 21:57:32.051093 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051081    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.051522 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051505    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.051606 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051510    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 21:57:32.051714 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.051691    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.052259 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.052156    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.052259 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.052175    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.052259 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.052246    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 21:57:32.052437 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.052281    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.052437 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.052363    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 21:57:32.053358 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053337    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 21:57:32.053358 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053356    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 21:57:32.053494 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053411    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.053494 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053450    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 21:57:32.053494 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053492    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053505    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053555    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vsl4h\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053564    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053569    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053592    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053592    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t5j6q\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053608    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053613    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053638    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 21:57:32.053695 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053662    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.054174 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053845    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 21:57:32.054174 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053899    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 21:57:32.054174 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053915    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.054174 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053924    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jdvwg\""
Nov 02 21:57:32.054174 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053947    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6vj4v\""
Nov 02 21:57:32.054420 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.053344    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.054840 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.054827    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:32.054933 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.054870    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:32.054933 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.054905    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 21:57:32.056256 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.056230    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 21:57:32.056336 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.056277    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 21:57:32.060568 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.060518    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xxf9z\""
Nov 02 21:57:32.060664 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.060518    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqhdf\""
Nov 02 21:57:32.060664 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.060596    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b8hrl\""
Nov 02 21:57:32.064899 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.064880    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 21:57:32.070612 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.070596    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 21:57:32.070694 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.070647    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 21:57:32.070746 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.070733    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 21:57:32.126649 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.126649 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qkb\" (UniqueName: \"kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb\") pod \"kas-connection-checker-685895c7c9-q9m4v\" (UID: \"0530c38c-3055-48b0-a829-ad3068c97528\") " pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr94q\" (UniqueName: \"kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q\") pod \"kas-connection-checker-685895c7c9-x8bf8\" (UID: \"3a2a4927-992c-4f11-9f42-81405452f431\") " pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.126786    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.126853    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.626831335 +0000 UTC m=+31.046901766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.126968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.127117 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2n8\" (UniqueName: \"kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8\") pod \"network-check-source-6f78d59f99-cw95x\" (UID: \"cdee2231-8f9e-486e-a740-051acb7b66f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.127679 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.127815    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.127902    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.627886287 +0000 UTC m=+31.047956718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:32.128131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.127926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhs5p\" (UniqueName: \"kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcsq\" (UniqueName: \"kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmgw\" (UniqueName: \"kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.128450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x84j\" (UniqueName: \"kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhv2\" (UniqueName: \"kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5br8\" (UniqueName: \"kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8\") pod \"kas-connection-checker-685895c7c9-plm4n\" (UID: \"b103cd80-a503-475f-b9a8-f448fbd0af73\") " pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.128722 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb25\" (UniqueName: \"kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.128958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhx6\" (UniqueName: \"kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bp6\" (UniqueName: \"kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.129094 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mndw\" (UniqueName: \"kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5w6\" (UniqueName: \"kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6\") pod \"volume-data-source-validator-58c9f5f46d-grv5x\" (UID: \"fbd984d3-01fe-4b43-9ac1-4259e6c39faa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.129107    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:32.129582 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.129585    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:32.130141 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.129650    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.629633151 +0000 UTC m=+31.049703584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:32.130141 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.130141 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.129996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.130481 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.130457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.131157 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.131128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.131471 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.131366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.131471 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.130829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.131471 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.131450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.131661 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.131602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.131921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.131902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.132299 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.132274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.133282 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.133252    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.135359 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.135337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qkb\" (UniqueName: \"kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb\") pod \"kas-connection-checker-685895c7c9-q9m4v\" (UID: \"0530c38c-3055-48b0-a829-ad3068c97528\") " pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:32.137746 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.137728    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr94q\" (UniqueName: \"kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q\") pod \"kas-connection-checker-685895c7c9-x8bf8\" (UID: \"3a2a4927-992c-4f11-9f42-81405452f431\") " pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:32.142385 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.142359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhs5p\" (UniqueName: \"kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.144460 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.144374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhx6\" (UniqueName: \"kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.144460 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.144417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb25\" (UniqueName: \"kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.145718 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.144828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhv2\" (UniqueName: \"kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.145718 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.144955    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.145718 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.145162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5br8\" (UniqueName: \"kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8\") pod \"kas-connection-checker-685895c7c9-plm4n\" (UID: \"b103cd80-a503-475f-b9a8-f448fbd0af73\") " pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:32.145718 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.145194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5w6\" (UniqueName: \"kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6\") pod \"volume-data-source-validator-58c9f5f46d-grv5x\" (UID: \"fbd984d3-01fe-4b43-9ac1-4259e6c39faa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:32.145718 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.145200    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.146385 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.146343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bp6\" (UniqueName: \"kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.229882 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.229850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.229882 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.229888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.229926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.229953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.229981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230034    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230098    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230110    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:32.230131 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2n8\" (UniqueName: \"kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8\") pod \"network-check-source-6f78d59f99-cw95x\" (UID: \"cdee2231-8f9e-486e-a740-051acb7b66f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230155    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.730139389 +0000 UTC m=+31.150209801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230189    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230256    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.73023886 +0000 UTC m=+31.150309275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230289    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.730274986 +0000 UTC m=+31.150345413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230320    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.730299254 +0000 UTC m=+31.150369683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcsq\" (UniqueName: \"kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.230489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmgw\" (UniqueName: \"kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x84j\" (UniqueName: \"kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230596    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.230728    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:32.730711822 +0000 UTC m=+31.150782236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : secret "router-metrics-certs-default" not found
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.230967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mndw\" (UniqueName: \"kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.231183 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.231921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.231921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.231921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231419    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.231921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.231451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.233287 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.233261    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.233703 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.233678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.233854 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.233833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.234024 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.234004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.234110 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.234091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.234332 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.234318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.239447 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.239420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.239447 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.239436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcsq\" (UniqueName: \"kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.239640 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.239524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.239730 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.239713    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2n8\" (UniqueName: \"kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8\") pod \"network-check-source-6f78d59f99-cw95x\" (UID: \"cdee2231-8f9e-486e-a740-051acb7b66f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:32.240120 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.240096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x84j\" (UniqueName: \"kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.240359 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.240340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mndw\" (UniqueName: \"kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.241159 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.241141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmgw\" (UniqueName: \"kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.248045 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.248024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:32.256672 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.256652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:32.272244 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.272216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:32.281201 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.281167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:32.288400 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.288374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:32.296752 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.296578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:32.309436 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.309400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:32.327798 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.326903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:32.351816 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.351546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:32.373797 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.368603    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:32.544901 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.544874    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plm4n"]
Nov 02 21:57:32.549196 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.549169    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x8bf8"]
Nov 02 21:57:32.556701 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.556676    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2a4927_992c_4f11_9f42_81405452f431.slice/crio-2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2 WatchSource:0}: Error finding container 2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2: Status 404 returned error can't find the container with id 2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2
Nov 02 21:57:32.559303 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.559283    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q9m4v"]
Nov 02 21:57:32.564831 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.564778    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0530c38c_3055_48b0_a829_ad3068c97528.slice/crio-f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487 WatchSource:0}: Error finding container f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487: Status 404 returned error can't find the container with id f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487
Nov 02 21:57:32.579593 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.579571    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jx4w4"]
Nov 02 21:57:32.587419 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.587387    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61cf27a_32bb_4a64_9ef4_56badad04b6d.slice/crio-fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9 WatchSource:0}: Error finding container fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9: Status 404 returned error can't find the container with id fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9
Nov 02 21:57:32.616842 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.616671    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"]
Nov 02 21:57:32.619695 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.619655    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd984d3_01fe_4b43_9ac1_4259e6c39faa.slice/crio-595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53 WatchSource:0}: Error finding container 595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53: Status 404 returned error can't find the container with id 595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53
Nov 02 21:57:32.636153 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.636118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:32.636303 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.636284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:32.636374 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.636358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:32.636520 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.636503    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:32.636594 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.636525    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:32.636654 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.636595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.636576903 +0000 UTC m=+32.056647329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:32.637023 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.637003    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:32.637085 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.637061    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.637045 +0000 UTC m=+32.057115424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:32.637143 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.637118    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:32.637197 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.637153    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.637141893 +0000 UTC m=+32.057212321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:32.651296 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.651190    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"]
Nov 02 21:57:32.655093 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.655049    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c60948_0876_4c3c_822e_a954bf3f02fb.slice/crio-f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16 WatchSource:0}: Error finding container f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16: Status 404 returned error can't find the container with id f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16
Nov 02 21:57:32.679969 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.679916    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"]
Nov 02 21:57:32.682716 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.682693    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418c8848_c1f8_449c_9406_f6c70830cde5.slice/crio-5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973 WatchSource:0}: Error finding container 5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973: Status 404 returned error can't find the container with id 5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973
Nov 02 21:57:32.721854 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.721829    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"]
Nov 02 21:57:32.726394 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.725996    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"]
Nov 02 21:57:32.731413 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.730202    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec070a08_538c_46a0_8f70_bf23a1468b0c.slice/crio-1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451 WatchSource:0}: Error finding container 1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451: Status 404 returned error can't find the container with id 1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451
Nov 02 21:57:32.736930 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.736903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:32.737025 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.736948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.737025 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.737000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:32.737171 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737056    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:32.737171 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737070    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:32.737171 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737074    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.737056963 +0000 UTC m=+32.157127387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:32.737171 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737124    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.737107391 +0000 UTC m=+32.157177808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:32.737171 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737126    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737176    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.737164635 +0000 UTC m=+32.157235059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : secret "router-metrics-certs-default" not found
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.737198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.737246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737338    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737346    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737366    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.73735851 +0000 UTC m=+32.157428933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:32.737394 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:32.737393    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:33.737384349 +0000 UTC m=+32.157454777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:32.742358 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:32.742325    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lknc7"]
Nov 02 21:57:32.745646 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:32.745620    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10347cb_927a_4d45_a48f_d9bc553d68db.slice/crio-04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88 WatchSource:0}: Error finding container 04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88: Status 404 returned error can't find the container with id 04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88
Nov 02 21:57:33.075857 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.075777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:33.075857 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.075814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:33.076009 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.075780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:33.079659 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.079634    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 21:57:33.079790 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.079667    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9dzjg\""
Nov 02 21:57:33.079790 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.079670    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6t4jd\""
Nov 02 21:57:33.079790 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.079729    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 21:57:33.080032 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.080015    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjf48\""
Nov 02 21:57:33.381083 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.381045    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" event={"ID":"ec070a08-538c-46a0-8f70-bf23a1468b0c","Type":"ContainerStarted","Data":"1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451"}
Nov 02 21:57:33.383808 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.383774    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm" event={"ID":"418c8848-c1f8-449c-9406-f6c70830cde5","Type":"ContainerStarted","Data":"5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973"}
Nov 02 21:57:33.385833 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.385804    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x" event={"ID":"fbd984d3-01fe-4b43-9ac1-4259e6c39faa","Type":"ContainerStarted","Data":"595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53"}
Nov 02 21:57:33.389271 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.388662    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x" event={"ID":"cdee2231-8f9e-486e-a740-051acb7b66f3","Type":"ContainerStarted","Data":"5c96d1062d6c45551daa8d33314270d5c53eb55bfe8e99896f92d3a5621acfaa"}
Nov 02 21:57:33.390295 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.390273    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2" event={"ID":"f7c60948-0876-4c3c-822e-a954bf3f02fb","Type":"ContainerStarted","Data":"f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16"}
Nov 02 21:57:33.393297 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.393273    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q9m4v" event={"ID":"0530c38c-3055-48b0-a829-ad3068c97528","Type":"ContainerStarted","Data":"6234841ff0f029b6b3b553cb35cea32f856eab5960e6701be62d04cd8ff4a8de"}
Nov 02 21:57:33.393409 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.393320    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q9m4v" event={"ID":"0530c38c-3055-48b0-a829-ad3068c97528","Type":"ContainerStarted","Data":"f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487"}
Nov 02 21:57:33.396046 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.396015    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x8bf8" event={"ID":"3a2a4927-992c-4f11-9f42-81405452f431","Type":"ContainerStarted","Data":"0ffb7c613751d685e9044a2ce56a6154509cbded8a9215d3e4afb109d743a9fa"}
Nov 02 21:57:33.396046 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.396051    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x8bf8" event={"ID":"3a2a4927-992c-4f11-9f42-81405452f431","Type":"ContainerStarted","Data":"2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2"}
Nov 02 21:57:33.398190 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.398163    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-plm4n" event={"ID":"b103cd80-a503-475f-b9a8-f448fbd0af73","Type":"ContainerStarted","Data":"4918e1890868e9b9899f4e51862ae3e28b004369189e56124927392ed2d6eeb6"}
Nov 02 21:57:33.398281 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.398203    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-plm4n" event={"ID":"b103cd80-a503-475f-b9a8-f448fbd0af73","Type":"ContainerStarted","Data":"79844bfc01cac03e50a3bc0f68776ba267cd6b306a29f0781657a2098138b7f7"}
Nov 02 21:57:33.402158 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.402131    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lknc7" event={"ID":"c10347cb-927a-4d45-a48f-d9bc553d68db","Type":"ContainerStarted","Data":"04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88"}
Nov 02 21:57:33.404514 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.404464    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jx4w4" event={"ID":"c61cf27a-32bb-4a64-9ef4-56badad04b6d","Type":"ContainerStarted","Data":"fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9"}
Nov 02 21:57:33.436481 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.434631    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-q9m4v" podStartSLOduration=444.434612243 podStartE2EDuration="7m24.434612243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:33.414449312 +0000 UTC m=+31.834519746" watchObservedRunningTime="2026-11-02 21:57:33.434612243 +0000 UTC m=+31.854682676"
Nov 02 21:57:33.454684 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.453871    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-x8bf8" podStartSLOduration=444.453852332 podStartE2EDuration="7m24.453852332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:33.452613608 +0000 UTC m=+31.872684048" watchObservedRunningTime="2026-11-02 21:57:33.453852332 +0000 UTC m=+31.873922766"
Nov 02 21:57:33.454684 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.454440    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-plm4n" podStartSLOduration=444.454430175 podStartE2EDuration="7m24.454430175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:33.436804846 +0000 UTC m=+31.856875278" watchObservedRunningTime="2026-11-02 21:57:33.454430175 +0000 UTC m=+31.874500609"
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.647334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.647491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.647572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.647692    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.647706    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.647762    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.647744455 +0000 UTC m=+34.067814880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.648179    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.648233    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.648217993 +0000 UTC m=+34.068288428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.648290    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:33.648431 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.648321    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.648310972 +0000 UTC m=+34.068381397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.748056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.748129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.748174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.748205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:33.748258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.748456    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.748519    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.748502119 +0000 UTC m=+34.168572536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : secret "router-metrics-certs-default" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.748923    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.748980    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.748963538 +0000 UTC m=+34.169033954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749039    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.749060852 +0000 UTC m=+34.169131264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749131    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749141    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749171    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.749160613 +0000 UTC m=+34.169231039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:33.749321 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:33.749232    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:35.749223146 +0000 UTC m=+34.169293557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:35.671248 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.671206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:35.671803 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.671309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:35.671803 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.671378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:35.671803 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.671712    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:35.671803 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.671778    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.671759593 +0000 UTC m=+38.091830021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:35.672025 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.671990    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:35.672025 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.672004    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:35.672114 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.672051    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.672036346 +0000 UTC m=+38.092106772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:35.672114 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.672063    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:35.672221 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.672132    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.672113423 +0000 UTC m=+38.092183849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:35.773084 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:35.773084 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:35.773325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:35.773325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:35.773325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:35.773325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:35.773325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.773280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:35.773612 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.773400    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:35.773612 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.773459    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.7734422 +0000 UTC m=+38.193512611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:35.773889 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.773867    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:35.773960 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.773894    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:35.773960 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.773944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.773927355 +0000 UTC m=+38.193997767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:35.774069 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774025    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.774013578 +0000 UTC m=+38.194084042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:35.774226 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774210    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:35.774311 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774259    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.774245277 +0000 UTC m=+38.194315912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:35.774779 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774761    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:57:35.774857 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774816    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:58:07.77480089 +0000 UTC m=+66.194871304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : secret "metrics-daemon-secret" not found
Nov 02 21:57:35.774913 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774891    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 21:57:35.774966 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:35.774928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.774916943 +0000 UTC m=+38.194987355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : secret "router-metrics-certs-default" not found
Nov 02 21:57:35.779552 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.777616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:35.806136 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.806102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:35.975560 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.975469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:35.987292 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:35.987260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:36.104477 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.104445    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvq2f"]
Nov 02 21:57:36.112568 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:36.112513    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d39c020_6a26_47a2_8f57_1655acdc05f0.slice/crio-ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf WatchSource:0}: Error finding container ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf: Status 404 returned error can't find the container with id ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf
Nov 02 21:57:36.124138 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.123887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:36.419305 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.419267    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x" event={"ID":"fbd984d3-01fe-4b43-9ac1-4259e6c39faa","Type":"ContainerStarted","Data":"c31720035c44962874a5dd3f0f8103ec5f7ec2f55bfe74514a852fd933be7c0e"}
Nov 02 21:57:36.424372 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.424277    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvq2f" event={"ID":"9d39c020-6a26-47a2-8f57-1655acdc05f0","Type":"ContainerStarted","Data":"ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf"}
Nov 02 21:57:36.459324 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.459263    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x" podStartSLOduration=421.66553897 podStartE2EDuration="7m4.459241593s" totalImagesPullingTime="2.793702623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:36.434635442 +0000 UTC m=+34.854705876" watchObservedRunningTime="2026-11-02 21:57:36.459241593 +0000 UTC m=+34.879312027"
Nov 02 21:57:36.462164 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:36.461139    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbg45"]
Nov 02 21:57:36.463620 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:36.463575    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8031ebee_f5f9_4efa_acac_f56c132e93cf.slice/crio-fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8 WatchSource:0}: Error finding container fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8: Status 404 returned error can't find the container with id fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8
Nov 02 21:57:37.430090 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.430052    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbg45" event={"ID":"8031ebee-f5f9-4efa-acac-f56c132e93cf","Type":"ContainerStarted","Data":"fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8"}
Nov 02 21:57:37.438774 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.438738    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jx4w4" event={"ID":"c61cf27a-32bb-4a64-9ef4-56badad04b6d","Type":"ContainerStarted","Data":"2fe34936e4418a13b49cd0e09f5d641ec87a868a09f5474f9687b863a7e93615"}
Nov 02 21:57:37.459637 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.459583    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-jx4w4" podStartSLOduration=421.171167276 podStartE2EDuration="7m5.459562933s" totalImagesPullingTime="4.288395657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:37.458954971 +0000 UTC m=+35.879025406" watchObservedRunningTime="2026-11-02 21:57:37.459562933 +0000 UTC m=+35.879633368"
Nov 02 21:57:37.492064 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.492022    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdmzs"]
Nov 02 21:57:37.530398 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.529605    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdmzs"]
Nov 02 21:57:37.530398 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.529734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.615786 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.611758    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 21:57:37.615786 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.611872    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v8fd\""
Nov 02 21:57:37.615786 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.611898    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 21:57:37.700713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.694874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.700713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.694961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpbc\" (UniqueName: \"kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.700713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.695011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.700713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.695064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.700713 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.695136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796145 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.796106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.796214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796327 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.796314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796575 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.796382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpbc\" (UniqueName: \"kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796575 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.796420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.796575 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:37.796546    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:57:37.796721 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:37.796615    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:38.296596293 +0000 UTC m=+36.716666719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : configmap "kube-rbac-proxy" not found
Nov 02 21:57:37.796941 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:37.796923    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:37.796992 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:37.796980    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:38.296964567 +0000 UTC m=+36.717034997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:37.797332 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.797304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.797430 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.797385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:37.806434 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:37.806363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpbc\" (UniqueName: \"kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:38.300696 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.300607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:38.300696 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.300662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:38.300895 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:38.300741    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:57:38.300895 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:38.300828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.300807351 +0000 UTC m=+37.720877775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : configmap "kube-rbac-proxy" not found
Nov 02 21:57:38.300895 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:38.300752    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:38.301054 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:38.300903    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:39.300885274 +0000 UTC m=+37.720955773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:38.346101 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.346062    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"]
Nov 02 21:57:38.365619 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.365561    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"]
Nov 02 21:57:38.365771 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.365688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:38.446878 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.446854    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/0.log"
Nov 02 21:57:38.447325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.446895    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec070a08-538c-46a0-8f70-bf23a1468b0c" containerID="94d642e700de68866753c550e4110f2c7b47922193f1b4130599b63b17b836d1" exitCode=255
Nov 02 21:57:38.447325 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.446993    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" event={"ID":"ec070a08-538c-46a0-8f70-bf23a1468b0c","Type":"ContainerDied","Data":"94d642e700de68866753c550e4110f2c7b47922193f1b4130599b63b17b836d1"}
Nov 02 21:57:38.447432 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.447362    2576 scope.go:122] "RemoveContainer" containerID="94d642e700de68866753c550e4110f2c7b47922193f1b4130599b63b17b836d1"
Nov 02 21:57:38.449904 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.449874    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm" event={"ID":"418c8848-c1f8-449c-9406-f6c70830cde5","Type":"ContainerStarted","Data":"cc253ed491962efa5978b168408a483c6a0c59b32212af68f3b4bbf2788e225f"}
Nov 02 21:57:38.452971 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.452945    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2" event={"ID":"f7c60948-0876-4c3c-822e-a954bf3f02fb","Type":"ContainerStarted","Data":"e2d2ae9667d7eaaca744425047dca706a06e0cc378e7bc9b653da7306a57be2f"}
Nov 02 21:57:38.456076 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.455998    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 21:57:38.456242 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.456222    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g9f57\""
Nov 02 21:57:38.456369 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.456280    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:38.502424 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.502153    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2" podStartSLOduration=419.660094392 podStartE2EDuration="7m4.502134568s" totalImagesPullingTime="4.842040176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:38.500861185 +0000 UTC m=+36.920931620" watchObservedRunningTime="2026-11-02 21:57:38.502134568 +0000 UTC m=+36.922205001"
Nov 02 21:57:38.502424 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.502255    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm" podStartSLOduration=416.339915932 podStartE2EDuration="7m1.50224932s" totalImagesPullingTime="5.162333388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:38.482016056 +0000 UTC m=+36.902086495" watchObservedRunningTime="2026-11-02 21:57:38.50224932 +0000 UTC m=+36.922319754"
Nov 02 21:57:38.502651 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.502461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vcr\" (UniqueName: \"kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr\") pod \"migrator-66df5d496b-xbm66\" (UID: \"786d8c8c-f90c-468b-8833-b51c7e4965b8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:38.604435 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.604398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vcr\" (UniqueName: \"kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr\") pod \"migrator-66df5d496b-xbm66\" (UID: \"786d8c8c-f90c-468b-8833-b51c7e4965b8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:38.614783 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.614756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vcr\" (UniqueName: \"kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr\") pod \"migrator-66df5d496b-xbm66\" (UID: \"786d8c8c-f90c-468b-8833-b51c7e4965b8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:38.712468 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:38.712422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:39.116043 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.115958    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"]
Nov 02 21:57:39.123765 ip-10-0-131-57 kubenswrapper[2576]: W1102 21:57:39.121129    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786d8c8c_f90c_468b_8833_b51c7e4965b8.slice/crio-8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982 WatchSource:0}: Error finding container 8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982: Status 404 returned error can't find the container with id 8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982
Nov 02 21:57:39.312495 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.312452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:39.312686 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.312519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:39.312686 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.312646    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:57:39.312686 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.312680    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:39.312880 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.312717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:41.31270145 +0000 UTC m=+39.732771879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : configmap "kube-rbac-proxy" not found
Nov 02 21:57:39.312880 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.312732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:41.312725991 +0000 UTC m=+39.732796401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:39.457482 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.457402    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:39.457967 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.457883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/0.log"
Nov 02 21:57:39.457967 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.457920    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec070a08-538c-46a0-8f70-bf23a1468b0c" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888" exitCode=255
Nov 02 21:57:39.458075 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.458013    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" event={"ID":"ec070a08-538c-46a0-8f70-bf23a1468b0c","Type":"ContainerDied","Data":"89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"}
Nov 02 21:57:39.458139 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.458081    2576 scope.go:122] "RemoveContainer" containerID="94d642e700de68866753c550e4110f2c7b47922193f1b4130599b63b17b836d1"
Nov 02 21:57:39.458379 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.458347    2576 scope.go:122] "RemoveContainer" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"
Nov 02 21:57:39.458650 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.458623    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:39.462072 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.461605    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x" event={"ID":"cdee2231-8f9e-486e-a740-051acb7b66f3","Type":"ContainerStarted","Data":"a9d5c4ef5d0da894435535bc0ae92989415bbe7d0ac3e74b784d610b7528affa"}
Nov 02 21:57:39.463603 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.463578    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lknc7" event={"ID":"c10347cb-927a-4d45-a48f-d9bc553d68db","Type":"ContainerStarted","Data":"9ad8310b3bba3f777638c8b5331984850fb6e8a2b5dee5384dc241547c6f490b"}
Nov 02 21:57:39.479371 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.479338    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66" event={"ID":"786d8c8c-f90c-468b-8833-b51c7e4965b8","Type":"ContainerStarted","Data":"8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982"}
Nov 02 21:57:39.494714 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.494657    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lknc7" podStartSLOduration=2.8089193630000002 podStartE2EDuration="8.494639902s" totalImagesPullingTime="5.685720539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:39.49350531 +0000 UTC m=+37.913575745" watchObservedRunningTime="2026-11-02 21:57:39.494639902 +0000 UTC m=+37.914710336"
Nov 02 21:57:39.510158 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.510097    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x" podStartSLOduration=412.391304254 podStartE2EDuration="6m58.51008068s" totalImagesPullingTime="6.118776426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:39.508916118 +0000 UTC m=+37.928986553" watchObservedRunningTime="2026-11-02 21:57:39.51008068 +0000 UTC m=+37.930151113"
Nov 02 21:57:39.717754 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.717660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:39.717921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.717754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:39.717921 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.717808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:39.717921 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717856    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:39.717921 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717900    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:39.717921 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717919    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:39.718177 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717926    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:39.718177 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.717914797 +0000 UTC m=+46.137985208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:39.718177 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.717956048 +0000 UTC m=+46.138026473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:39.718177 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.717992    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.717982658 +0000 UTC m=+46.138053075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:39.819023 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.818990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:39.819270 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819150    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 21:57:39.819270 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.819220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:39.819389 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.819275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:39.819389 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.819311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:39.819389 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:39.819341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:39.819389 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819346    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819394    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.819382129 +0000 UTC m=+46.239452562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819438    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819460    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.819451716 +0000 UTC m=+46.239522164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819470    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.819464939 +0000 UTC m=+46.239535349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819481    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.819475418 +0000 UTC m=+46.239545828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : secret "router-metrics-certs-default" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819521    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819559    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:39.819616 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:39.819604    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.819589418 +0000 UTC m=+46.239659836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:40.478378 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:40.478300    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbg45" event={"ID":"8031ebee-f5f9-4efa-acac-f56c132e93cf","Type":"ContainerStarted","Data":"6c807f0c93c95282e3d2629927367554aee666348f05f5e6629f5fc0b2feebf5"}
Nov 02 21:57:40.478378 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:40.478364    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:40.479833 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:40.479809    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:40.480182 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:40.480160    2576 scope.go:122] "RemoveContainer" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"
Nov 02 21:57:40.480391 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:40.480367    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:40.493928 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:40.493876    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kbg45" podStartSLOduration=34.830301079 podStartE2EDuration="38.493858871s" totalImagesPullingTime="3.663557792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:40.492698654 +0000 UTC m=+38.912769087" watchObservedRunningTime="2026-11-02 21:57:40.493858871 +0000 UTC m=+38.913929305"
Nov 02 21:57:41.334099 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:41.334002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:41.334099 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:41.334060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:41.334290 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:41.334105    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:57:41.334290 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:41.334184    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.334168024 +0000 UTC m=+43.754238460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : configmap "kube-rbac-proxy" not found
Nov 02 21:57:41.334290 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:41.334216    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:41.334290 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:41.334266    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.334255011 +0000 UTC m=+43.754325436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:41.485618 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:41.485510    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66" event={"ID":"786d8c8c-f90c-468b-8833-b51c7e4965b8","Type":"ContainerStarted","Data":"8e96a2de7f6bb2243f50fbe1f08e6c2aec7ebf3c3795c33850b8ba59783ec397"}
Nov 02 21:57:42.327489 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.327450    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:42.327700 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.327561    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:42.327954 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.327937    2576 scope.go:122] "RemoveContainer" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"
Nov 02 21:57:42.328172 ip-10-0-131-57 kubenswrapper[2576]: E1102 21:57:42.328149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:42.493237 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.493198    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvq2f" event={"ID":"9d39c020-6a26-47a2-8f57-1655acdc05f0","Type":"ContainerStarted","Data":"46ac26ade6c505ef7534f6ce8c3dcbb7ecf332b9e57a22605b5ca816e3b85e34"}
Nov 02 21:57:42.507497 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.507449    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bvq2f" podStartSLOduration=34.291876445 podStartE2EDuration="40.507435973s" totalImagesPullingTime="6.215559528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:42.506934794 +0000 UTC m=+40.927005227" watchObservedRunningTime="2026-11-02 21:57:42.507435973 +0000 UTC m=+40.927506633"
Nov 02 21:57:42.507756 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.507733    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66" podStartSLOduration=2.745146128 podStartE2EDuration="4.507725409s" totalImagesPullingTime="1.762579281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:41.500980243 +0000 UTC m=+39.921050676" watchObservedRunningTime="2026-11-02 21:57:42.507725409 +0000 UTC m=+40.927795851"
Nov 02 21:57:42.634450 ip-10-0-131-57 kubenswrapper[2576]: I1102 21:57:42.634248    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:57:42.634408 ip-10-0-131-57 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 21:57:42.644971 ip-10-0-131-57 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 21:57:42.645139 ip-10-0-131-57 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 21:57:42.645189 ip-10-0-131-57 systemd[1]: kubelet.service: Consumed 3.061s CPU time.
Nov 02 21:57:42.661172 ip-10-0-131-57 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 21:57:42.720913 ip-10-0-131-57 kubenswrapper[4929]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:42.720913 ip-10-0-131-57 kubenswrapper[4929]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 21:57:42.720913 ip-10-0-131-57 kubenswrapper[4929]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:42.720913 ip-10-0-131-57 kubenswrapper[4929]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:57:42.723046 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.722955    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.723046 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723045    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723050    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723054    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723057    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723060    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723064    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723067    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723070    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723073    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723076    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723079    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723082    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723085    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723087    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723090    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723093    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723095    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723098    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723101    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723103    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.723125 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723106    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723109    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723111    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723114    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723116    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723119    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723126    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723130    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723133    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723136    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723139    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723141    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723144    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723147    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723150    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723152    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723155    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723157    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723160    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723162    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723165    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723167    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723170    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723173    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723175    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723178    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723181    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723183    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723186    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723188    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723191    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723194    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723197    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723200    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723203    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723206    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723208    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723211    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723214    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.724113 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723217    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723220    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723222    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723226    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723228    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723231    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723235    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723239    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723243    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723246    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723249    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723252    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723255    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723258    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723261    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723263    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723266    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723268    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723271    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.724595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723274    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723276    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723279    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723282    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723285    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723287    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723291    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723294    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723297    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723299    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723302    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723305    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723307    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723310    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723313    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723318    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723322    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723325    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723329    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.725095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723332    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723334    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723337    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723340    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723342    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723345    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723347    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.723350    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724154    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724164    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724167    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724169    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724172    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724176    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724179    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724183    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724187    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724190    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724193    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.725617 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724196    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724200    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724203    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724206    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724209    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724212    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724214    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724217    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724220    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724223    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724226    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724229    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724231    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724234    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724237    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724241    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724245    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724248    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724251    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.726097 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724254    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724257    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724260    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724262    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724265    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724268    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724270    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724273    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724276    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724279    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724282    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724285    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724288    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724290    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724293    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724296    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724299    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724302    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724304    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724307    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.726592 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724310    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724312    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724316    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724319    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724322    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724325    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724327    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724330    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724333    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724336    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724339    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724341    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724344    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724347    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724349    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724352    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724355    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724357    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724360    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724363    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.727074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724365    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724368    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724370    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724374    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724376    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724379    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724382    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724384    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724387    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724390    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724393    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724395    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724398    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724401    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724403    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724406    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724410    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724412    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724415    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724417    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.727595 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724421    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724423    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724426    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724429    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724432    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724434    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724437    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724439    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724442    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724445    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724447    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724450    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724453    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724455    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724459    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.724463    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725001    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725007    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725010    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.728088 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725013    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725016    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725019    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725022    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725025    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725027    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725031    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725034    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725037    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725041    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725044    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725047    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725050    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725052    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725055    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725058    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725061    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725063    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725066    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.728618 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725069    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725073    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725077    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725080    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725083    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725086    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725088    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725091    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725094    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725097    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725100    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725103    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725106    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725109    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725112    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725115    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725118    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725120    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725123    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725126    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.729146 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725128    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725131    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725134    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725137    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725140    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725143    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725145    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725148    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725150    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725153    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725156    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725158    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725161    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725164    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725166    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725171    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725173    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725176    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725179    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725181    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.729773 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725184    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725187    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725189    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725192    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725197    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725200    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725202    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725205    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725208    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725210    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725213    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725216    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725219    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725221    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725224    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725226    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725230    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725232    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725235    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.730263 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725238    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725241    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725245    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725249    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725252    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725255    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725258    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725260    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725263    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725266    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725268    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725272    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725275    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725277    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725280    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725283    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725286    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725289    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725292    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.730738 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725295    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725297    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725300    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725302    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725305    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.725307    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725407    4929 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725414    4929 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725427    4929 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725432    4929 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725436    4929 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725440    4929 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725444    4929 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725449    4929 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725452    4929 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725455    4929 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725459    4929 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725462    4929 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725465    4929 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725468    4929 flags.go:64] FLAG: --cgroup-root=""
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725471    4929 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725474    4929 flags.go:64] FLAG: --client-ca-file=""
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725478    4929 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 21:57:42.731243 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725481    4929 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725488    4929 flags.go:64] FLAG: --cluster-domain=""
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725491    4929 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725495    4929 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725498    4929 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725502    4929 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725506    4929 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725513    4929 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725517    4929 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725528    4929 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725547    4929 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725552    4929 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725557    4929 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725562    4929 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725566    4929 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725572    4929 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725575    4929 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725578    4929 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725581    4929 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725585    4929 flags.go:64] FLAG: --enable-server="true"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725588    4929 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725601    4929 flags.go:64] FLAG: --event-burst="100"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725605    4929 flags.go:64] FLAG: --event-qps="50"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725608    4929 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 21:57:42.731828 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725611    4929 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725614    4929 flags.go:64] FLAG: --eviction-hard=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725619    4929 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725622    4929 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725625    4929 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725628    4929 flags.go:64] FLAG: --eviction-soft=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725632    4929 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725635    4929 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725638    4929 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725641    4929 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725644    4929 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725647    4929 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725650    4929 flags.go:64] FLAG: --feature-gates=""
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725657    4929 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725660    4929 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725665    4929 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725668    4929 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725671    4929 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725674    4929 flags.go:64] FLAG: --help="false"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725677    4929 flags.go:64] FLAG: --hostname-override="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725680    4929 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725683    4929 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725686    4929 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 21:57:42.732866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725691    4929 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725695    4929 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725698    4929 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725701    4929 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725704    4929 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725707    4929 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725710    4929 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725720    4929 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725723    4929 flags.go:64] FLAG: --kube-reserved=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725726    4929 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725729    4929 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725732    4929 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725735    4929 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725738    4929 flags.go:64] FLAG: --lock-file=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725741    4929 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725744    4929 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725748    4929 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725753    4929 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725756    4929 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725760    4929 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725763    4929 flags.go:64] FLAG: --logging-format="text"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725766    4929 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725769    4929 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725773    4929 flags.go:64] FLAG: --manifest-url=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725777    4929 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 21:57:42.733468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725781    4929 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725784    4929 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725788    4929 flags.go:64] FLAG: --max-pods="110"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725792    4929 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725795    4929 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725798    4929 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725804    4929 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725810    4929 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725813    4929 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725817    4929 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725825    4929 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725828    4929 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725831    4929 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725834    4929 flags.go:64] FLAG: --pod-cidr=""
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725838    4929 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725846    4929 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725850    4929 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725853    4929 flags.go:64] FLAG: --port="10250"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725856    4929 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725859    4929 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08690e48e0b48737e"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725862    4929 flags.go:64] FLAG: --qos-reserved=""
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725865    4929 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725869    4929 flags.go:64] FLAG: --register-node="true"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725872    4929 flags.go:64] FLAG: --register-with-taints=""
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725876    4929 flags.go:64] FLAG: --registry-burst="10"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725879    4929 flags.go:64] FLAG: --registry-qps="5"
Nov 02 21:57:42.734093 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725882    4929 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725885    4929 flags.go:64] FLAG: --reserved-memory=""
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725889    4929 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725893    4929 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725896    4929 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725900    4929 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725904    4929 flags.go:64] FLAG: --runonce="false"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725907    4929 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725910    4929 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725913    4929 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725916    4929 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725919    4929 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725923    4929 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725927    4929 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725931    4929 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725934    4929 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725937    4929 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725940    4929 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725943    4929 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725946    4929 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725949    4929 flags.go:64] FLAG: --system-cgroups=""
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725952    4929 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725963    4929 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725967    4929 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725970    4929 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 21:57:42.734775 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725979    4929 flags.go:64] FLAG: --tls-min-version=""
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725982    4929 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725984    4929 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725987    4929 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725990    4929 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725994    4929 flags.go:64] FLAG: --v="2"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.725998    4929 flags.go:64] FLAG: --version="false"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.726002    4929 flags.go:64] FLAG: --vmodule=""
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.726006    4929 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.726009    4929 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726182    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726188    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726195    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726199    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726202    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726206    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726209    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726212    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726214    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726217    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726221    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.735371 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726224    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726227    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726230    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726233    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726235    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726238    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726241    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726244    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726247    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726255    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726259    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726261    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726264    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726267    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726270    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726272    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726275    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726278    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726287    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726290    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.735923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726292    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726295    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726298    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726303    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726306    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726309    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726311    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726314    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726317    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726319    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726322    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726327    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726330    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726332    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726335    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726338    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726340    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726343    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726346    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726348    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.736419 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726351    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726354    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726362    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726365    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726368    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726370    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726373    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726376    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726379    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726381    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726384    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726386    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726389    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726392    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726394    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726399    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726402    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726404    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726407    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726410    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.736987 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726412    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726415    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726418    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726422    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726425    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726427    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726430    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726433    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726436    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726438    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726441    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726444    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726446    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726449    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726452    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726460    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726463    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726466    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726469    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.737492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726472    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726475    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726477    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726480    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726482    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726485    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726488    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726490    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726494    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726497    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726500    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726502    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726505    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726508    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726510    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726513    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.738206 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.726554    4929 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726598    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726603    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726607    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726610    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726613    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726616    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726619    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726622    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726625    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726628    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726630    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726633    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726642    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726645    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726648    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726651    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726653    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726656    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.738699 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726658    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726661    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726664    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726667    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726669    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726674    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726677    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726680    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726682    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726685    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726688    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726690    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726693    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726696    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726699    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726702    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726704    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726707    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726710    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726712    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.739213 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726715    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726718    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726720    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726723    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726726    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726728    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726731    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726741    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726744    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726746    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726749    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726752    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726754    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726757    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726759    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726762    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726765    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726769    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726771    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.739769 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726774    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726777    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726780    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726782    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726785    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726787    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726790    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726793    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726796    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726798    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726801    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726803    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726806    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726809    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726812    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726814    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726817    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726821    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726825    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.740292 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726829    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726832    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726841    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726844    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726847    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726850    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726852    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726855    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726857    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726860    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726863    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726867    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726869    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726873    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726875    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726878    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726881    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726883    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726886    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726889    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.740846 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726892    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726895    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726897    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726900    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726902    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726905    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726908    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726911    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726914    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.726917    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.730870    4929 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.730886    4929 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.730901    4929 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.730906    4929 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730975    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730982    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730986    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730990    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730993    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.741351 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730996    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.730999    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731002    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731005    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731008    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731011    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731014    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731017    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731020    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731023    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731025    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731028    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731031    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731033    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731036    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731039    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731042    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731044    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731047    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731050    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.741993 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731053    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731056    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731059    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731062    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731066    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731068    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731071    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731074    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731077    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731079    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731082    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731085    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731087    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731090    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731093    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731095    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731098    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731100    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731103    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.742492 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731106    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731108    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731111    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731114    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731116    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731119    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731122    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731125    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731128    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731130    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731133    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731135    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731138    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731141    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731144    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731147    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731150    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731152    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731155    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731158    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.742972 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731161    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731163    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731166    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731169    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731172    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731174    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731177    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731180    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731183    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731185    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731188    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731191    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731193    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731196    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731199    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731201    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731204    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731207    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731209    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.743447 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731212    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731215    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731218    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731221    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731223    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731226    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731229    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731231    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731234    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731237    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731240    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731242    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731246    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731249    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731252    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731255    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731257    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731260    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731263    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731267    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.743923 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731270    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731273    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731275    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.731303    4929 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731478    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731483    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731486    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731489    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731492    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731495    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731498    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731500    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731503    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731506    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731509    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731511    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.744440 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731514    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731517    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731519    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731522    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731525    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731528    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731544    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731548    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731551    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731555    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731558    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731561    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731564    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731567    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731569    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731572    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731575    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731578    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731580    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.744848 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731583    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731586    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731589    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731593    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731597    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731664    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731672    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.731677    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732211    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732231    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732238    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732243    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732248    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732252    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732257    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732262    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732267    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732272    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732278    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732283    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.745307 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732293    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732298    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732304    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732328    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732334    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732339    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732345    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732350    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732355    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732359    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732363    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732368    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732377    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732382    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732386    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732391    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732395    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732400    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732404    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.745814 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732412    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732417    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732443    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732448    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732452    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732456    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732466    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732471    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732476    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732480    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732484    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732489    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732493    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732514    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732520    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732524    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732529    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732560    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732565    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732575    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.746297 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732579    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732584    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732588    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732593    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732598    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732602    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732607    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732611    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732617    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732636    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732641    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732645    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732656    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732660    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732665    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.732671    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.747013 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.732757    4929 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733030    4929 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733069    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733237    4929 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733242    4929 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733246    4929 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733250    4929 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733253    4929 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733256    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733259    4929 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733262    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733265    4929 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733268    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733271    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733273    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733276    4929 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733279    4929 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733282    4929 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733285    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733288    4929 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:57:42.747441 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733291    4929 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733294    4929 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733297    4929 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733300    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733303    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733306    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733309    4929 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733312    4929 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733315    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733318    4929 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733321    4929 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733324    4929 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733327    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733331    4929 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733334    4929 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733337    4929 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733342    4929 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733344    4929 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733347    4929 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:57:42.747957 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733350    4929 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733353    4929 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733356    4929 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733358    4929 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733362    4929 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733364    4929 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733367    4929 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733370    4929 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733373    4929 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733376    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733379    4929 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733382    4929 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733385    4929 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733388    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733391    4929 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733393    4929 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733396    4929 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733399    4929 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733403    4929 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733406    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:57:42.748422 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733412    4929 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733415    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733418    4929 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733420    4929 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733423    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733426    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733429    4929 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733432    4929 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733434    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733437    4929 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733440    4929 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733444    4929 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733448    4929 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733451    4929 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733454    4929 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733457    4929 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733460    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733463    4929 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733465    4929 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733468    4929 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:57:42.748951 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733472    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733475    4929 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733478    4929 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733481    4929 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733484    4929 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733487    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733490    4929 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733493    4929 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733496    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733499    4929 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733504    4929 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733508    4929 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733512    4929 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733515    4929 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733517    4929 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733528    4929 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733551    4929 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733554    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733557    4929 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:57:42.749438 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733560    4929 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733563    4929 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733566    4929 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733568    4929 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733571    4929 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733574    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733577    4929 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733580    4929 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:42.733582    4929 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.733677    4929 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.734687    4929 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.734740    4929 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.734915    4929 server.go:1014] "Starting client certificate rotation"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.735056    4929 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.735105    4929 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:52:01 +0000 UTC" deadline="2028-04-22 05:14:58.804874569 +0000 UTC"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.735152    4929 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12871h17m16.069725635s"
Nov 02 21:57:42.749940 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.735571    4929 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:57:42.750608 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.736828    4929 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:57:42.750608 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.737968    4929 log.go:25] "Validated CRI v1 runtime API"
Nov 02 21:57:42.750608 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.739797    4929 log.go:25] "Validated CRI v1 image API"
Nov 02 21:57:42.750608 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.740453    4929 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 21:57:42.750608 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.742478    4929 fs.go:135] Filesystem UUIDs: map[43cc123c-c5bf-47b5-b122-9ee5f7ab2cac:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7e36c000-ef26-46a5-97b9-d3379b87e499:/dev/nvme0n1p3]
Nov 02 21:57:42.750841 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.742497    4929 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c96d1062d6c45551daa8d33314270d5c53eb55bfe8e99896f92d3a5621acfaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c96d1062d6c45551daa8d33314270d5c53eb55bfe8e99896f92d3a5621acfaa/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79844bfc01cac03e50a3bc0f68776ba267cd6b306a29f0781657a2098138b7f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79844bfc01cac03e50a3bc0f68776ba267cd6b306a29f0781657a2098138b7f7/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0530c38c-3055-48b0-a829-ad3068c97528/volumes/kubernetes.io~projected/kube-api-access-z5qkb:{mountpoint:/var/lib/kubelet/pods/0530c38c-3055-48b0-a829-ad3068c97528/volumes/kubernetes.io~projected/kube-api-access-z5qkb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a2571db-e8bb-431c-a74d-dea6f9a261cc/volumes/kubernetes.io~projected/kube-api-access-xwpsh:{mountpoint:/var/lib/kubelet/pods/0a2571db-e8bb-431c-a74d-dea6f9a261cc/volumes/kubernetes.io~projected/kube-api-access-xwpsh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3/volumes/kubernetes.io~projected/kube-api-access-l427q:{mountpoint:/var/lib/kubelet/pods/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3/volumes/kubernetes.io~projected/kube-api-access-l427q major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~projected/kube-api-access-xjcsq:{mountpoint:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~projected/kube-api-access-xjcsq major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/default-certificate major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~projected/kube-api-access-22xbp:{mountpoint:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~projected/kube-api-access-22xbp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36174814-afd3-42b6-a555-75c832ebacf5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/36174814-afd3-42b6-a555-75c832ebacf5/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a2a4927-992c-4f11-9f42-81405452f431/volumes/kubernetes.io~projected/kube-api-access-dr94q:{mountpoint:/var/lib/kubelet/pods/3a2a4927-992c-4f11-9f42-81405452f431/volumes/kubernetes.io~projected/kube-api-access-dr94q major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~projected/kube-api-access-2x84j:{mountpoint:/var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~projected/kube-api-access-2x84j major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a19a824-6d01-4257-a6c5-90c2d7f0c649/volumes/kubernetes.io~projected/kube-api-access-zzjs5:{mountpoint:/var/lib/kubelet/pods/6a19a824-6d01-4257-a6c5-90c2d7f0c649/volumes/kubernetes.io~projected/kube-api-access-zzjs5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/730c3bbf-87e5-4378-a258-a990d610143c/volumes/kubernetes.io~projected/kube-api-access-lxpbc:{mountpoint:/var/lib/kubelet/pods/730c3bbf-87e5-4378-a258-a990d610143c/volumes/kubernetes.io~projected/kube-api-access-lxpbc major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/786d8c8c-f90c-468b-8833-b51c7e4965b8/volumes/kubernetes.io~projected/kube-api-access-r4vcr:{mountpoint:/var/lib/kubelet/pods/786d8c8c-f90c-468b-8833-b51c7e4965b8/volumes/kubernetes.io~projected/kube-api-access-r4vcr major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8031ebee-f5f9-4efa-acac-f56c132e93cf/volumes/kubernetes.io~projected/kube-api-access-pn7wv:{mountpoint:/var/lib/kubelet/pods/8031ebee-f5f9-4efa-acac-f56c132e93cf/volumes/kubernetes.io~projected/kube-api-access-pn7wv major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d91b2f9-d641-449a-9bae-601f629cc281/volumes/kubernetes.io~projected/kube-api-access-r98n2:{mountpoint:/var/lib/kubelet/pods/8d91b2f9-d641-449a-9bae-601f629cc281/volumes/kubernetes.io~projected/kube-api-access-r98n2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d39c020-6a26-47a2-8f57-1655acdc05f0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9d39c020-6a26-47a2-8f57-1655acdc05f0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a113cb77-4c64-47ea-8d09-723c2146038f/volumes/kubernetes.io~projected/kube-api-access-8tb25:{mountpoint:/var/lib/kubelet/pods/a113cb77-4c64-47ea-8d09-723c2146038f/volumes/kubernetes.io~projected/kube-api-access-8tb25 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b103cd80-a503-475f-b9a8-f448fbd0af73/volumes/kubernetes.io~projected/kube-api-access-f5br8:{mountpoint:/var/lib/kubelet/pods/b103cd80-a503-475f-b9a8-f448fbd0af73/volumes/kubernetes.io~projected/kube-api-access-f5br8 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2f05bd7-b4eb-4df9-af57-fb0277839916/volumes/kubernetes.io~projected/kube-api-access-tb7mh:{mountpoint:/var/lib/kubelet/pods/b2f05bd7-b4eb-4df9-af57-fb0277839916/volumes/kubernetes.io~projected/kube-api-access-tb7mh major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b911671e-958f-441e-92cc-bf023d0757a7/volumes/kubernetes.io~projected/kube-api-access-8grxx:{mountpoint:/var/lib/kubelet/pods/b911671e-958f-441e-92cc-bf023d0757a7/volumes/kubernetes.io~projected/kube-api-access-8grxx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf18c217-668e-4f22-8bb3-ea403e33e962/volumes/kubernetes.io~projected/kube-api-access-kt568:{mountpoint:/var/lib/kubelet/pods/bf18c217-668e-4f22-8bb3-ea403e33e962/volumes/kubernetes.io~projected/kube-api-access-kt568 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~projected/kube-api-access-bbmgw:{mountpoint:/var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~projected/kube-api-access-bbmgw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~projected/kube-api-access-dfhv2:{mountpoint:/var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~projected/kube-api-access-dfhv2 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/kube-api-access-gwp6z:{mountpoint:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/kube-api-access-gwp6z major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdee2231-8f9e-486e-a740-051acb7b66f3/volumes/kubernetes.io~projected/kube-api-access-dr2n8:{mountpoint:/var/lib/kubelet/pods/cdee2231-8f9e-486e-a740-051acb7b66f3/volumes/kubernetes.io~projected/kube-api-access-dr2n8 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7c34109-7853-45e8-a813-29b53a775844/volumes/kubernetes.io~projected/kube-api-access-pjhx6:{mountpoint:/var/lib/kubelet/pods/d7c34109-7853-45e8-a813-29b53a775844/volumes/kubernetes.io~projected/kube-api-access-pjhx6 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~projected/kube-api-access-4lxsr:{mountpoint:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~projected/kube-api-access-4lxsr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/kube-api-access-nhglq:{mountpoint:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/kube-api-access-nhglq major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~projected/kube-api-access-bhs5p:{mountpoint:/var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~projected/kube-api-access-bhs5p major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~projected/kube-api-access-87bp6:{mountpoint:/var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~projected/kube-api-access-87bp6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbd984d3-01fe-4b43-9ac1-4259e6c39faa/volumes/kubernetes.io~projected/kube-api-access-9q5w6:{mountpoint:/var/lib/kubelet/pods/fbd984d3-01fe-4b43-9ac1-4259e6c39faa/volumes/kubernetes.io~projected/kube-api-access-9q5w6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feae0ada-a336-48c6-a163-7a968597ad33/volumes/kubernetes.io~projected/kube-api-access-2mndw:{mountpoint:/var/lib/kubelet/pods/feae0ada-a336-48c6-a163-7a968597ad33/volumes/kubernetes.io~projected/kube-api-access-2mndw major:0 minor:226 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ced1202cd8b38f9bb7ae88582214ffcde3bb1104c82ba5132d8a6838538c60b4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/02b0b054a952d74897e5540e08c1904d01674e1a25919b079dcc08580232bb83/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3ac54a11666d01a870b1c6757eca920fcb9e7c586a32ceb473daa04643bb8415/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/db09dd0bd92631e864c05eb35db4f4d1a73a567dd222afdef85d4f38142edbe3/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c8cca2ed0a4a2eda472bddb6ba3010680c038a38bd91230e36b5a434c452df8d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ac93ea17cbbb65b86559bc046517da58272b21a58fdc75286794b5f729176bcc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d4c477f375d212d615c58d062a08213a4cc7dbce3438159ccf364ce439f50c91/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e2a0fe424c8c942e499e609001427e94d056cb2e2a3bf0fdeadfa08d11aa3393/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/17193387345e0813bc16a6bcc1c8fcaf0b617e1479beb2c1f2322a6174cb5acc/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/14149fa30509dcfb7770205b90ae36ad8345aecef446b243e2966bf4c7702762/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/8e43002d64fa715643955ae8abb1955e17a1951f65f1baa79814a36694443c68/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8f32a01eae05de520e5ea44b2f91639dc56347d3872acd0980da0fdc257ce422/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/86b6085cbb766904f27ef1c66c21200b00494d813d52b3aa5fe0046338ac4271/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/624d9cf07478ea338917aa4d38918e851bed5d1782d6af9a64d1e24fd24e4023/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bed9bc536d2acdbdd7e29ae7b8f866ecd60d9aa7d5492070f044a31baead5d85/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0611344f2859dba55bf9b636727065b6272c35334b71ce2459ee0aa4ff5fad44/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9faf29e2c9f73b3e71170f2e50cc7d1ffe38c4eadabb1088b1386c0697825c1d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/51c9609c25de2bc982f77a14de5a52c9be05100d200c1ed61f9749fa3f524db9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/81e274a21bc4f0fb1a9fdb9f9da68fcaa302e2671e4ede5f81029fe0012c4046/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f749981b7f7fe66c8a53da760050b004f9a751cf7024fd3800f42997c0dde687/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2e2c0802cab27ed93f03c35b950908fade7fb35f244774d8efc924e030da9f52/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5d806f47d7833335928cfc01acc9849f2c35511580458155089748c907a8ecce/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/03c4fb58b18c5f22b16a9de5010af043b9e5022fe7abb88ed7b6b7a104a8974a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/18b43f85624967ed0bfc56903cba0f2c1c0467afcf387e19dac3bff4ec22ce3c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fa93e4d949242ba968dfbee8cc9180cf21e34d49f6d49ab18a24b71643ccebc5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/62da4e35ad5f003c5e5d7fbf1adabc5a866680a9b39abe129735db6560003bb4/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/35decdf392b2ad9c8bb95fdfba2168ff740ae17a3ab256c46348ed3c48002f68/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/4352e1d9e4b7427f5b21b70f88fd7af9a2621582952cc5594a85e48349dad9d6/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9ac1a3eb41d5d21ede1b034d770354fb8c5adbd368fe02972518fbb07380e04d/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/9d5b3db551edbbc07fe9540fc21913904327ef519df79bd696b59b79b5ec6eb0/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5f4289ed389f1129e963e85da3b9f054723f8bec6ddd7ef522bab9ade407ae0d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/44a0c5917b00beae13eb3388ffaa95cbe69e63d778e5178f120509388cbbdeec/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/8c1a7d417f531dc5c4d25b525821a4b5b9098a11a11f7bb5d469711ba7a6ff8b/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/8706659e7513ded2a41d2400fc1b5db84ad3e4f1e01bd9a3d35c4841a31880fc/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/108fea902739f6d3395166e3c894015556c0155869bdf72dd3a45e3ea932c511/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/d84ca91b5f79d3f498f19c76cdaab5748783ba6f8ff99c4570d1faee5654fd58/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/89854eafd8557b10f2f9641180aa219e9b05c1a5994cac89f4fc153714460cc3/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/74f83d491fde7db15911e1199b9b909bc0a4770feafe55b031954cd31e2b4ffb/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/91b0c53531daab3e4a01be3cd18027d9ba46f45b97aa356c1026fc9e937ec2e3/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/067a594cb41437bd44ab51aae44a155a56016dd6734b8ffe6e25edb7c2443c33/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/e53945709a4bb16e6fc8e212f2b912d14f0c105445b8eac6b7153a0e0a83947c/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/2802b8384696830a22589a73b8d8dba6785e9849e0e9a9161b46613484ba41a7/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/820cdcbe88ef57e358d27094014a43215d9d839ad99f7a22c57a654158edb788/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/7f036c1428932ab4b166994a8a89c5240bcd450223bcf32f9f8ba11902f4b977/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/730cfd4d482606042e37286cb783ac9ed557e66f79921ed25953df2e6a52cfe8/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/bfeb5c7a0964cb94be4029d2535326ea7bf4fcdccfe8e50900c59e377cd0cde1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f849590a51b18d98212a8ce7e657f323e0b1ba575215205e2cfe0573a3d693a2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c43b5508fd15e7041d006c8808c5bb0a3b581c2d48575b272212c0b2b5f37266/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/9e041b3a87b51592dc7c336455160908afb38531c9a4475eaab0d7876f270ab8/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/31c54406cb7db8d1fb7a9aa2fc00f314a5c067d9a07888f42fbedf3596f73b6d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1d79351cc54c3eecd87ed4adaf1e6d93fe99aa09264899985142f7b2e19856f0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d81cd34265c53c20b3ef8ff9d68dc72e544f81cd68c60f067a9e5a91a5b09314/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/20a7b97ff9eea1c8735f8a0895cacf615f934865edeb1aadd9c6d9c40d9b716c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/91a71aad5e01fa12707ee9aae3dc7b0b1538ad63c15dc6892b6f0238e56b98b4/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/abd0acacb928b402130904e2fd8de4a011b16e722d7f4167c789dde4f38b9dff/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/3dea175f8b57291ef4269b9b7a58af0561787d7864bffc2eac5c0daff99654e0/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 02 21:57:42.751766 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.751156    4929 manager.go:217] Machine: {Timestamp:2026-11-02 21:57:42.750299103 +0000 UTC m=+0.058389900 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099378 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d46bdafeb3e8652b609d9602dcebc SystemUUID:ec2d46bd-afeb-3e86-52b6-09d9602dcebc BootID:c4c6a56d-9f8d-4b3d-a016-040a7e464a15 Filesystems:[{Device:/run/containers/storage/overlay-containers/fe81463264416c71c3ac618d5a514ece662a7972f63f0859d90a04767f06d2a8/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~projected/kube-api-access-4lxsr DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cea1ef54394011ac7b489c93ec580b4ffe13c06e9ee57fe93ad0819d69bc2eb7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3/volumes/kubernetes.io~projected/kube-api-access-l427q DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e8bd11be04a9939066e3693b3b908a1a5199d011d65e8a84b8b996b148dbc3a/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~projected/kube-api-access-87bp6 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a113cb77-4c64-47ea-8d09-723c2146038f/volumes/kubernetes.io~projected/kube-api-access-8tb25 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b103cd80-a503-475f-b9a8-f448fbd0af73/volumes/kubernetes.io~projected/kube-api-access-f5br8 DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7c34109-7853-45e8-a813-29b53a775844/volumes/kubernetes.io~projected/kube-api-access-pjhx6 DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cdee2231-8f9e-486e-a740-051acb7b66f3/volumes/kubernetes.io~projected/kube-api-access-dr2n8 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05ea5929da6aac6b29884f1e3988f25cea9c08bc3b1edd178d9b73c25783ed99/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~projected/kube-api-access-bbmgw DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/418c8848-c1f8-449c-9406-f6c70830cde5/volumes/kubernetes.io~projected/kube-api-access-2x84j DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f728086d11e6daa56aa89353e156220980c1ab22be87c0431eecff8719298487/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~projected/kube-api-access-22xbp DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/299a0e837ff0f79da3cf3f4e7aa2ee2b4ad20b8eb6447ca625e4b639e715bdf0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/kube-api-access-nhglq DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79844bfc01cac03e50a3bc0f68776ba267cd6b306a29f0781657a2098138b7f7/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a19a824-6d01-4257-a6c5-90c2d7f0c649/volumes/kubernetes.io~projected/kube-api-access-zzjs5 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c60948-0876-4c3c-822e-a954bf3f02fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bbd63d7cc4e06138a1f47a678b9d9bad83bb647fc239cc0189f4b4f79111982/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09977011b72e54a4ea8e85df66eebbcdb39d99c8677d8c7efbe7092ecc1167e6/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~projected/kube-api-access-bhs5p DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~projected/kube-api-access-dfhv2 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5469b17a1378c7f6e4a4149d5fc8d267b30cc7597921bd2fd36acdc138661973/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/278e0ebb48cbbf48381f4523bdfe8f10930b0472dceede40d6bd347b999f4cf2/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6173aba8b1453a8266612c4a0045e70cf51e27941f55027a6742ea04c620d21b/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec070a08-538c-46a0-8f70-bf23a1468b0c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc9a1325a3232a1df31aa529f77e0cdc2adfa18ffb468b06fcc2390c33e860a9/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba117582313b7c8657d298dddc511649754445cb00be5dd7cfcf3af8a856f0cf/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2f05bd7-b4eb-4df9-af57-fb0277839916/volumes/kubernetes.io~projected/kube-api-access-tb7mh DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/342f55ac-11b3-442a-beb6-f2aadf8aa899/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0530c38c-3055-48b0-a829-ad3068c97528/volumes/kubernetes.io~projected/kube-api-access-z5qkb DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/730c3bbf-87e5-4378-a258-a990d610143c/volumes/kubernetes.io~projected/kube-api-access-lxpbc DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2925725a7d6ce625ba6bc247ea9cefeccd3f50bb883686b09318b83f5ecd6ac2/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/36174814-afd3-42b6-a555-75c832ebacf5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes/kubernetes.io~projected/kube-api-access-gwp6z DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c10347cb-927a-4d45-a48f-d9bc553d68db/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0f5aae5-63bc-4e64-a357-1b046c4fc470/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a91d4c6e873ea85977636deb39d819bcc974fc65c587747e78de682334a7d86/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d39c020-6a26-47a2-8f57-1655acdc05f0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0a2571db-e8bb-431c-a74d-dea6f9a261cc/volumes/kubernetes.io~projected/kube-api-access-xwpsh DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d91b2f9-d641-449a-9bae-601f629cc281/volumes/kubernetes.io~projected/kube-api-access-r98n2 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c61cf27a-32bb-4a64-9ef4-56badad04b6d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~projected/kube-api-access-xjcsq DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/feae0ada-a336-48c6-a163-7a968597ad33/volumes/kubernetes.io~projected/kube-api-access-2mndw DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2612587be71cee0582eae76db75f89651af309fd196e76c099fef3354ea3e16/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/786d8c8c-f90c-468b-8833-b51c7e4965b8/volumes/kubernetes.io~projected/kube-api-access-r4vcr DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b911671e-958f-441e-92cc-bf023d0757a7/volumes/kubernetes.io~projected/kube-api-access-8grxx DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/327d452e-cee7-493d-bd9a-81dc5b5b42e2/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a40007aa575ce3fe5453b58609c1590b524698b7246ecdcbdcc22f801b70451/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c96d1062d6c45551daa8d33314270d5c53eb55bfe8e99896f92d3a5621acfaa/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/595ddc085ae69cd35ee576042a61a6816add1d0086bb4d8663a845dcd0ef7c53/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8031ebee-f5f9-4efa-acac-f56c132e93cf/volumes/kubernetes.io~projected/kube-api-access-pn7wv DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf18c217-668e-4f22-8bb3-ea403e33e962/volumes/kubernetes.io~projected/kube-api-access-kt568 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbd984d3-01fe-4b43-9ac1-4259e6c39faa/volumes/kubernetes.io~projected/kube-api-access-9q5w6 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04fb46459cafb6b683933ccba41349c14f6141f822aa210ab50de361d12e0c88/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a2a4927-992c-4f11-9f42-81405452f431/volumes/kubernetes.io~projected/kube-api-access-dr94q DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04fb46459cafb6b MacAddress:ba:0f:16:c1:a2:06 Speed:10000 Mtu:8901} {Name:1a40007aa575ce3 MacAddress:2e:0e:30:7a:cd:00 Speed:10000 Mtu:8901} {Name:2925725a7d6ce62 MacAddress:c2:6e:27:e5:e8:f0 Speed:10000 Mtu:8901} {Name:5469b17a1378c7f MacAddress:8a:d6:e2:82:c7:4a Speed:10000 Mtu:8901} {Name:595ddc085ae69cd MacAddress:46:f0:99:17:54:ca Speed:10000 Mtu:8901} {Name:5c96d1062d6c455 MacAddress:96:1b:e5:ad:0c:b4 Speed:10000 Mtu:8901} {Name:79844bfc01cac03 MacAddress:de:b7:f5:9b:2d:72 Speed:10000 Mtu:8901} {Name:8bbd63d7cc4e061 MacAddress:3e:45:04:94:08:31 Speed:10000 Mtu:8901} {Name:ba117582313b7c8 MacAddress:b2:2e:e9:e1:fc:c9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:61:b0:3b Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:30:6d:01:e7:43 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:61:b0:3b Speed:0 Mtu:9001} {Name:f2612587be71cee MacAddress:b6:86:72:de:ad:0a Speed:10000 Mtu:8901} {Name:f728086d11e6daa MacAddress:1e:e2:cc:38:cd:93 Speed:10000 Mtu:8901} {Name:fc9a1325a3232a1 MacAddress:0e:fa:62:3f:fb:72 Speed:10000 Mtu:8901} {Name:fe81463264416c7 MacAddress:aa:a9:2b:b4:65:21 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:9a:dc:78:04:ec:56 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:14:6e:45:cc:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 21:57:42.752052 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.751772    4929 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 21:57:42.752052 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.751832    4929 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 21:57:42.752220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752181    4929 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 21:57:42.752375 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752222    4929 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-57.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 21:57:42.752451 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752380    4929 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 21:57:42.752451 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752393    4929 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 21:57:42.752451 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752411    4929 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:57:42.752451 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752438    4929 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:57:42.752658 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752468    4929 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 21:57:42.752731 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752718    4929 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 21:57:42.752854 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752843    4929 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 21:57:42.752907 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752891    4929 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 21:57:42.752907 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752905    4929 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 21:57:42.752996 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752919    4929 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 21:57:42.752996 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752933    4929 kubelet.go:405] "Adding apiserver pod source"
Nov 02 21:57:42.752996 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.752944    4929 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 21:57:42.753665 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.753649    4929 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 21:57:42.753665 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.753663    4929 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:57:42.754170 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.754153    4929 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 21:57:42.756307 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756293    4929 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 21:57:42.756307 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756308    4929 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:57:42.756404 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756395    4929 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 21:57:42.756647 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756592    4929 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 21:57:42.756647 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756622    4929 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 21:57:42.756730 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756720    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 21:57:42.756760 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756740    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 21:57:42.756760 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756751    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 21:57:42.756760 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756758    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756766    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756772    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756777    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756783    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756789    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756795    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756808    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 21:57:42.756840 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756831    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 21:57:42.757081 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756860    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 21:57:42.757081 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.756868    4929 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 21:57:42.760443 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.760429    4929 server.go:1263] "Started kubelet"
Nov 02 21:57:42.762746 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.760597    4929 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 21:57:42.762746 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.760594    4929 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 21:57:42.762746 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.760692    4929 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 21:57:42.762746 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.761288    4929 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 21:57:42.762746 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.762198    4929 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 21:57:42.761144 ip-10-0-131-57 systemd[1]: Started Kubernetes Kubelet.
Nov 02 21:57:42.766369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766193    4929 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 21:57:42.766369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766282    4929 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766688    4929 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766701    4929 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766763    4929 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:04 +0000 UTC" deadline="2028-05-24 23:34:34.637969704 +0000 UTC"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766804    4929 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13657h36m51.871169879s"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:42.766813    4929 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-57.us-east-2.compute.internal\" not found"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.766961    4929 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.767367    4929 factory.go:55] Registering systemd factory
Nov 02 21:57:42.769134 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.767384    4929 factory.go:223] Registration of the systemd container factory successfully
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770417    4929 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770608    4929 factory.go:153] Registering CRI-O factory
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770623    4929 factory.go:223] Registration of the crio container factory successfully
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770739    4929 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770759    4929 factory.go:103] Registering Raw factory
Nov 02 21:57:42.771079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.770773    4929 manager.go:1199] Started watching for new ooms in manager
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771175    4929 manager.go:319] Starting recovery of all containers
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771320    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771368    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec070a08-538c-46a0-8f70-bf23a1468b0c" volumeName="kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771384    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327d452e-cee7-493d-bd9a-81dc5b5b42e2" volumeName="kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771398    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771414    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="730c3bbf-87e5-4378-a258-a990d610143c" volumeName="kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771427    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2afa6f34-35a6-4bcd-939e-8083b7cb4fb3" volumeName="kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771440    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342f55ac-11b3-442a-beb6-f2aadf8aa899" volumeName="kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771455    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342f55ac-11b3-442a-beb6-f2aadf8aa899" volumeName="kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771471    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="418c8848-c1f8-449c-9406-f6c70830cde5" volumeName="kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771511    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a19a824-6d01-4257-a6c5-90c2d7f0c649" volumeName="kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771526    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f05bd7-b4eb-4df9-af57-fb0277839916" volumeName="kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771552    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771564    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771577    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36174814-afd3-42b6-a555-75c832ebacf5" volumeName="kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771592    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0f5aae5-63bc-4e64-a357-1b046c4fc470" volumeName="kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771604    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771620    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c60948-0876-4c3c-822e-a954bf3f02fb" volumeName="kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771647    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feae0ada-a336-48c6-a163-7a968597ad33" volumeName="kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771662    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f05bd7-b4eb-4df9-af57-fb0277839916" volumeName="kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771674    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b911671e-958f-441e-92cc-bf023d0757a7" volumeName="kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771686    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771698    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342f55ac-11b3-442a-beb6-f2aadf8aa899" volumeName="kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771724    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10347cb-927a-4d45-a48f-d9bc553d68db" volumeName="kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771746    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d39c020-6a26-47a2-8f57-1655acdc05f0" volumeName="kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771758    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a2a4927-992c-4f11-9f42-81405452f431" volumeName="kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771770    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="418c8848-c1f8-449c-9406-f6c70830cde5" volumeName="kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771781    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf18c217-668e-4f22-8bb3-ea403e33e962" volumeName="kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771793    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771811    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771823    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771836    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7c34109-7853-45e8-a813-29b53a775844" volumeName="kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771847    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771859    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feae0ada-a336-48c6-a163-7a968597ad33" volumeName="kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771871    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec070a08-538c-46a0-8f70-bf23a1468b0c" volumeName="kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771883    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec070a08-538c-46a0-8f70-bf23a1468b0c" volumeName="kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771894    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a19a824-6d01-4257-a6c5-90c2d7f0c649" volumeName="kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771905    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771917    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771929    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feae0ada-a336-48c6-a163-7a968597ad33" volumeName="kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771941    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771955    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7c34109-7853-45e8-a813-29b53a775844" volumeName="kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771966    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a2571db-e8bb-431c-a74d-dea6f9a261cc" volumeName="kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771978    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="730c3bbf-87e5-4378-a258-a990d610143c" volumeName="kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.771990    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f05bd7-b4eb-4df9-af57-fb0277839916" volumeName="kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772001    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f05bd7-b4eb-4df9-af57-fb0277839916" volumeName="kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772013    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e87a2020-99e4-4d23-bd7c-47ed49b6925e" volumeName="kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772027    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327d452e-cee7-493d-bd9a-81dc5b5b42e2" volumeName="kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772038    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772050    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772061    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772074    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec070a08-538c-46a0-8f70-bf23a1468b0c" volumeName="kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772088    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c60948-0876-4c3c-822e-a954bf3f02fb" volumeName="kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772102    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d91b2f9-d641-449a-9bae-601f629cc281" volumeName="kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772114    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10347cb-927a-4d45-a48f-d9bc553d68db" volumeName="kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772126    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdee2231-8f9e-486e-a740-051acb7b66f3" volumeName="kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772137    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0f5aae5-63bc-4e64-a357-1b046c4fc470" volumeName="kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772158    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c60948-0876-4c3c-822e-a954bf3f02fb" volumeName="kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772170    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a2571db-e8bb-431c-a74d-dea6f9a261cc" volumeName="kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772183    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0f5aae5-63bc-4e64-a357-1b046c4fc470" volumeName="kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772196    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="786d8c8c-f90c-468b-8833-b51c7e4965b8" volumeName="kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772208    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d36c5e6-09a7-4301-b6a1-db669e212d61" volumeName="kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772220    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8031ebee-f5f9-4efa-acac-f56c132e93cf" volumeName="kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772233    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36174814-afd3-42b6-a555-75c832ebacf5" volumeName="kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772245    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a19a824-6d01-4257-a6c5-90c2d7f0c649" volumeName="kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772264    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b103cd80-a503-475f-b9a8-f448fbd0af73" volumeName="kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772276    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772288    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0f5aae5-63bc-4e64-a357-1b046c4fc470" volumeName="kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772303    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a113cb77-4c64-47ea-8d09-723c2146038f" volumeName="kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772314    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0530c38c-3055-48b0-a829-ad3068c97528" volumeName="kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772325    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327d452e-cee7-493d-bd9a-81dc5b5b42e2" volumeName="kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772338    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="418c8848-c1f8-449c-9406-f6c70830cde5" volumeName="kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772349    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf18c217-668e-4f22-8bb3-ea403e33e962" volumeName="kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772360    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31455a-0dba-4b59-b254-39739cbe0790" volumeName="kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772372    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0f5aae5-63bc-4e64-a357-1b046c4fc470" volumeName="kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772388    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbd984d3-01fe-4b43-9ac1-4259e6c39faa" volumeName="kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772400    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2afa6f34-35a6-4bcd-939e-8083b7cb4fb3" volumeName="kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772415    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a113cb77-4c64-47ea-8d09-723c2146038f" volumeName="kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772427    4929 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61cf27a-32bb-4a64-9ef4-56badad04b6d" volumeName="kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772440    4929 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 21:57:42.772860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.772447    4929 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 21:57:42.776170 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.773936    4929 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:42.776170 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.773961    4929 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:42.776170 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.775386    4929 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 21:57:42.783755 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.783721    4929 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 21:57:42.786505 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.786483    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:42.789281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.789247    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:42.796631 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.796303    4929 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 21:57:42.796631 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.796330    4929 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 21:57:42.796631 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.796349    4929 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 21:57:42.796631 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:42.796387    4929 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 21:57:42.804012 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.803993    4929 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:57:42.812119 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.812086    4929 generic.go:356] "Generic (PLEG): container finished" podID="664740e8a5efdbcb60ae50b0e894ce2d" containerID="01709f200936b9e8f85ae1390c31c3ff25a48afba1c3ccb138a6f664fdc8d27e" exitCode=0
Nov 02 21:57:42.822821 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822788    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="5c9929ac0e73521660521a8bec2ae7d519b0c1f9897f31a135651d45e12c7082" exitCode=0
Nov 02 21:57:42.822821 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822823    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="85cbe655f6580206745845c32f5c48e4472b9efab22bdf0b07f84060d4f792b1" exitCode=0
Nov 02 21:57:42.822979 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822834    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3f0697db4bccbe41758964f4626ac82e5d4ef1dc3e8ffc173bb624c0c5da457a" exitCode=0
Nov 02 21:57:42.822979 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822843    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="bffe358a47a34b9a5e0c810a657f66e2cb4872d0c5f43a68da0c1500f62fac65" exitCode=0
Nov 02 21:57:42.822979 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822852    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3a6ab2e430674a2d393360d5f65d20c4d9f3273bc055c0a78da0168d13df1664" exitCode=0
Nov 02 21:57:42.822979 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.822860    4929 generic.go:356] "Generic (PLEG): container finished" podID="b2f05bd7-b4eb-4df9-af57-fb0277839916" containerID="3041ac3e94357d75e0b7b9dda4d4a6a8f7c5b3f2f3eccd7388e31567a1387b81" exitCode=0
Nov 02 21:57:42.847455 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.847427    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:42.847609 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.847460    4929 generic.go:356] "Generic (PLEG): container finished" podID="ec070a08-538c-46a0-8f70-bf23a1468b0c" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888" exitCode=255
Nov 02 21:57:42.861407 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.861378    4929 manager.go:324] Recovery completed
Nov 02 21:57:42.881454 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881429    4929 cpu_manager.go:226] "Starting" policy="none"
Nov 02 21:57:42.881454 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881448    4929 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 21:57:42.881649 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881467    4929 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 21:57:42.881649 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881619    4929 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 21:57:42.881649 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881631    4929 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 21:57:42.881649 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881649    4929 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 21:57:42.881757 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881655    4929 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 21:57:42.881757 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881660    4929 policy_none.go:50] "Start"
Nov 02 21:57:42.881757 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881668    4929 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 21:57:42.881757 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.881677    4929 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 21:57:42.890582 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.890525    4929 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 21:57:42.890582 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.890558    4929 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 21:57:42.890582 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.890567    4929 policy_none.go:44] "Start"
Nov 02 21:57:42.894919 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.894889    4929 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 21:57:42.896806 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:42.896787    4929 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 21:57:42.897422 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.897408    4929 manager.go:341] "Starting Device Plugin manager"
Nov 02 21:57:42.897912 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:42.897441    4929 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 21:57:42.897987 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.897915    4929 server.go:87] "Starting device plugin registration server"
Nov 02 21:57:42.898209 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898196    4929 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 21:57:42.898269 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898210    4929 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 21:57:42.898328 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898302    4929 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 21:57:42.898442 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898416    4929 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 21:57:42.898509 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898451    4929 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 21:57:42.898509 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898457    4929 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 21:57:42.898616 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.898495    4929 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:57:42.898441378Z","UUID":"d13dd052-cf43-4dee-ad45-db70ae190c12","Handler":null,"Name":"","Endpoint":""}
Nov 02 21:57:42.899855 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.899836    4929 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 21:57:42.899935 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.899865    4929 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 21:57:42.900182 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:42.900160    4929 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 21:57:42.999180 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:42.999138    4929 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:57:43.000420 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.000399    4929 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:57:43.000574 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.000429    4929 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:57:43.000574 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.000441    4929 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-57.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:57:43.000574 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.000482    4929 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.015653 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.015630    4929 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.015787 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.015702    4929 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.097265 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.097206    4929 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"]
Nov 02 21:57:43.097967 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.097919    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"d522352ece329563a2383a55c8031694","Type":"ContainerStarted","Data":"e63262c6d389993c19f572d542ab27a4249d5788e75ce020e25e7463ee779388"}
Nov 02 21:57:43.098024 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.097980    4929 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.098077 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.098060    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"d522352ece329563a2383a55c8031694","Type":"ContainerStarted","Data":"ab11acbf1900aa45103e8105e083003bbfa65fcd72aeb2f26bcdbd0433b4ab5f"}
Nov 02 21:57:43.098107 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.098088    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerStarted","Data":"1c2aeb157ffe4e5dee870b912471b88744a394ca71b13e5874b2988c682ae4ad"}
Nov 02 21:57:43.098148 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.098109    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerDied","Data":"01709f200936b9e8f85ae1390c31c3ff25a48afba1c3ccb138a6f664fdc8d27e"}
Nov 02 21:57:43.098148 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.098128    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal" event={"ID":"664740e8a5efdbcb60ae50b0e894ce2d","Type":"ContainerStarted","Data":"ed928a34f734100cb6e3a15a13e4d8bfc22bacaff0dc1b4f744c41dee1041817"}
Nov 02 21:57:43.098205 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.098160    4929 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.110274 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.110251    4929 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:43.110344 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.110300    4929 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.110793 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.110777    4929 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:43.110834 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.110817    4929 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.177214 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.177130    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.177214 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.177167    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.177214 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.177183    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277769 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277726    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277769 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277770    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277946 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277798    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277946 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277839    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d522352ece329563a2383a55c8031694-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"d522352ece329563a2383a55c8031694\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277946 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277852    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.277946 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.277894    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/664740e8a5efdbcb60ae50b0e894ce2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" (UID: \"664740e8a5efdbcb60ae50b0e894ce2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.753447 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.753408    4929 apiserver.go:52] "Watching apiserver"
Nov 02 21:57:43.814960 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.814928    4929 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 21:57:43.815567 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.815525    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-6rjz9","openshift-ovn-kubernetes/ovnkube-node-55qz7","openshift-dns/node-resolver-xcrch","kube-system/kas-connection-checker-685895c7c9-x8bf8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs","openshift-multus/multus-hclsq","openshift-cluster-node-tuning-operator/tuned-r9brh","openshift-network-console/networking-console-plugin-86c478bb88-vnfgs","kube-system/konnectivity-agent-kzxpt","openshift-insights/insights-operator-7c4986d449-jx4w4","openshift-image-registry/image-registry-998796474-kqqtd","kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-kbg45","kube-system/global-pull-secret-syncer-bvq2f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv","openshift-multus/network-metrics-daemon-nvvxb","openshift-console-operator/console-operator-7cd56b8dbc-jpvb6","openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w","openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2","openshift-dns/dns-default-kcsw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66","kube-system/kas-connection-checker-685895c7c9-plm4n","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x","openshift-ingress-canary/ingress-canary-lknc7","openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x","openshift-image-registry/image-registry-57bf9d85b7-pxw8s","kube-system/kas-connection-checker-685895c7c9-q9m4v","openshift-insights/insights-runtime-extractor-tdmzs","openshift-multus/multus-additional-cni-plugins-vxz77","openshift-ingress/router-default-8469b8f4f5-bf9r8","openshift-image-registry/node-ca-85npx"]
Nov 02 21:57:43.817283 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.817259    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.817933 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.817916    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.818698 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.818681    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.819799 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.819780    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.820108 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.820087    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.820244 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.820219    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:43.820338 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.820253    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.820338 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.820316    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 21:57:43.820443 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.820337    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.821022 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.821003    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:43.826516 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.826327    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 21:57:43.826516 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.826437    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 21:57:43.829321 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.829301    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.830361 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.830345    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.830907 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.830893    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.838343 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.838321    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 21:57:43.839169 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839150    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839176    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839187    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839203    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839222    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839269    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839281 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839269    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 21:57:43.839468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839292    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839468 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839309    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839564 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839487    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839564 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839493    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839564 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839545    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839673 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839593    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 21:57:43.839673 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839635    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839673 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839649    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839673 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839661    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839825 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839677    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 21:57:43.839825 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839682    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 21:57:43.839825 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839731    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839825 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839744    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 21:57:43.839825 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839767    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839969 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839876    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.839969 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839921    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.839969 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839950    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.840062 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839973    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.840062 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.839978    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.840480 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.840466    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.840480 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.840475    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 21:57:43.840556 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.840485    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.840556 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.840503    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841115 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841101    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841177 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841144    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 21:57:43.841177 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841170    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841202    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841206    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841224    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841259    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841271    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841275    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841281    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841285    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841296    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841314    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841326    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841333 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841286    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841341    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841326    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841331    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841345    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841363    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.841694 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.841413    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 21:57:43.844992 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.844978    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 21:57:43.847064 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.847051    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 21:57:43.850620 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.850604    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 21:57:43.850892 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.850876    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 21:57:43.851583 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.851570    4929 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.851695 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.851683    4929 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.864665 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.864640    4929 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:43.864809 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.864754    4929 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.864870 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.864812    4929 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:57:43.864870 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.864846    4929 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.us-east-2.compute.internal"
Nov 02 21:57:43.871545 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.871506    4929 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 21:57:43.881131 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881106    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb25\" (UniqueName: \"kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.881251 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881136    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.881251 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881155    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.881251 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881170    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.881251 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881191    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.881389 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881281    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.881389 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881348    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mndw\" (UniqueName: \"kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.881449 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881408    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grxx\" (UniqueName: \"kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:43.881449 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881429    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.881511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881445    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7c60948-0876-4c3c-822e-a954bf3f02fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.881511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881450    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.881511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881480    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.881676 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881529    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.881676 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881561    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-binary-copy\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.881779 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881669    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.881779 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881708    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.881779 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881727    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-service-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.881779 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881757    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881787    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vcr\" (UniqueName: \"kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr\") pod \"migrator-66df5d496b-xbm66\" (UID: \"786d8c8c-f90c-468b-8833-b51c7e4965b8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881831    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bp6\" (UniqueName: \"kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881857    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881892    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61cf27a-32bb-4a64-9ef4-56badad04b6d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881897    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.881983 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.881961    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d36c5e6-09a7-4301-b6a1-db669e212d61-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882016    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882046    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882080    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882113    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882139    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.882234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882181    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882209    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xbp\" (UniqueName: \"kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882266    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882306    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882333    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882343    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-config\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882359    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882400    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882423    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882449    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882475    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.882514 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882500    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882525    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882571    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5br8\" (UniqueName: \"kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8\") pod \"kas-connection-checker-685895c7c9-plm4n\" (UID: \"b103cd80-a503-475f-b9a8-f448fbd0af73\") " pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882572    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-tmp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882585    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-daemon-config\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882595    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x84j\" (UniqueName: \"kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882620    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882645    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882688    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882729    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882740    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf18c217-668e-4f22-8bb3-ea403e33e962-tmp-dir\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882771    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882827    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882848    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882915    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882946    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.882972    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.883041 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883038    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883049    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883087    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883105    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883121    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883110    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883158    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5w6\" (UniqueName: \"kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6\") pod \"volume-data-source-validator-58c9f5f46d-grv5x\" (UID: \"fbd984d3-01fe-4b43-9ac1-4259e6c39faa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883204    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883205    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883340    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883374    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883402    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883426    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883452    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883479    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr94q\" (UniqueName: \"kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q\") pod \"kas-connection-checker-685895c7c9-x8bf8\" (UID: \"3a2a4927-992c-4f11-9f42-81405452f431\") " pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883507    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2n8\" (UniqueName: \"kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8\") pod \"network-check-source-6f78d59f99-cw95x\" (UID: \"cdee2231-8f9e-486e-a740-051acb7b66f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883498    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/418c8848-c1f8-449c-9406-f6c70830cde5-config\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883563    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2f05bd7-b4eb-4df9-af57-fb0277839916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883596    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883650    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpbc\" (UniqueName: \"kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883714    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883736    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-tmp\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.883764 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883757    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883786    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-tuned\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883822    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt568\" (UniqueName: \"kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883866    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883886    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883903    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883918    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.883977    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qkb\" (UniqueName: \"kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb\") pod \"kas-connection-checker-685895c7c9-q9m4v\" (UID: \"0530c38c-3055-48b0-a829-ad3068c97528\") " pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884018    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884044    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884070    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884101    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884131    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884189    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884222    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884251    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884276    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884306    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61cf27a-32bb-4a64-9ef4-56badad04b6d-serving-cert\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884361    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884381    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884407    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884427    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884553    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.884601 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884587    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884628    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a2571db-e8bb-431c-a74d-dea6f9a261cc-iptables-alerter-script\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884640    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884677    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884725    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884746    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884766    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884782    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884803    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884827    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884860    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7mh\" (UniqueName: \"kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884941    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884974    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.884999    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885001    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36174814-afd3-42b6-a555-75c832ebacf5-konnectivity-ca\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885041    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885060    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/feae0ada-a336-48c6-a163-7a968597ad33-config-volume\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885073    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmgw\" (UniqueName: \"kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885151    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885189    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885214    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885253    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.885348 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885335    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885388    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885414    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cni-binary-copy\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885420    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885447    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885473    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l427q\" (UniqueName: \"kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885497    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885567    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxsr\" (UniqueName: \"kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885594    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhs5p\" (UniqueName: \"kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885627    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-config\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885702    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcsq\" (UniqueName: \"kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885731    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885764    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885821    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885871    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhx6\" (UniqueName: \"kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885905    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885948    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885967    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.885983    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886008    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886034    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886094    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886132    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-env-overrides\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886153    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feae0ada-a336-48c6-a163-7a968597ad33-tmp-dir\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886173    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886186    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886215    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886234    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886245    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c61cf27a-32bb-4a64-9ef4-56badad04b6d-snapshots\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886270    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886297    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886314    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886329    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.886369 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886383    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886415    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886440    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886457    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886516    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886566    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpsh\" (UniqueName: \"kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886594    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886621    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886711    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886746    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjs5\" (UniqueName: \"kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886757    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec070a08-538c-46a0-8f70-bf23a1468b0c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886772    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886809    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/730c3bbf-87e5-4378-a258-a990d610143c-data-volume\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886847    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a113cb77-4c64-47ea-8d09-723c2146038f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886895    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886913    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886931    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886949    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.886967    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhv2\" (UniqueName: \"kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.887116    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovnkube-script-lib\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.887452 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.887149    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-serviceca\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.888395 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.888371    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grxx\" (UniqueName: \"kubernetes.io/projected/b911671e-958f-441e-92cc-bf023d0757a7-kube-api-access-8grxx\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:43.888460 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.888444    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mndw\" (UniqueName: \"kubernetes.io/projected/feae0ada-a336-48c6-a163-7a968597ad33-kube-api-access-2mndw\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.889139 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889114    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vcr\" (UniqueName: \"kubernetes.io/projected/786d8c8c-f90c-468b-8833-b51c7e4965b8-kube-api-access-r4vcr\") pod \"migrator-66df5d496b-xbm66\" (UID: \"786d8c8c-f90c-468b-8833-b51c7e4965b8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbm66"
Nov 02 21:57:43.889229 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889118    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bp6\" (UniqueName: \"kubernetes.io/projected/f7c60948-0876-4c3c-822e-a954bf3f02fb-kube-api-access-87bp6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.889229 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889164    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xbp\" (UniqueName: \"kubernetes.io/projected/342f55ac-11b3-442a-beb6-f2aadf8aa899-kube-api-access-22xbp\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.889229 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889165    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb25\" (UniqueName: \"kubernetes.io/projected/a113cb77-4c64-47ea-8d09-723c2146038f-kube-api-access-8tb25\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.889476 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889453    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.889548 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.889489    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5br8\" (UniqueName: \"kubernetes.io/projected/b103cd80-a503-475f-b9a8-f448fbd0af73-kube-api-access-f5br8\") pod \"kas-connection-checker-685895c7c9-plm4n\" (UID: \"b103cd80-a503-475f-b9a8-f448fbd0af73\") " pod="kube-system/kas-connection-checker-685895c7c9-plm4n"
Nov 02 21:57:43.890046 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.890027    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5w6\" (UniqueName: \"kubernetes.io/projected/fbd984d3-01fe-4b43-9ac1-4259e6c39faa-kube-api-access-9q5w6\") pod \"volume-data-source-validator-58c9f5f46d-grv5x\" (UID: \"fbd984d3-01fe-4b43-9ac1-4259e6c39faa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-grv5x"
Nov 02 21:57:43.891971 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.891954    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpbc\" (UniqueName: \"kubernetes.io/projected/730c3bbf-87e5-4378-a258-a990d610143c-kube-api-access-lxpbc\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.892569 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.892551    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x84j\" (UniqueName: \"kubernetes.io/projected/418c8848-c1f8-449c-9406-f6c70830cde5-kube-api-access-2x84j\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.892885 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.892862    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt568\" (UniqueName: \"kubernetes.io/projected/bf18c217-668e-4f22-8bb3-ea403e33e962-kube-api-access-kt568\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.893032 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.893010    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2n8\" (UniqueName: \"kubernetes.io/projected/cdee2231-8f9e-486e-a740-051acb7b66f3-kube-api-access-dr2n8\") pod \"network-check-source-6f78d59f99-cw95x\" (UID: \"cdee2231-8f9e-486e-a740-051acb7b66f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cw95x"
Nov 02 21:57:43.893804 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.893788    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l427q\" (UniqueName: \"kubernetes.io/projected/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-kube-api-access-l427q\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.894145 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.894128    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7mh\" (UniqueName: \"kubernetes.io/projected/b2f05bd7-b4eb-4df9-af57-fb0277839916-kube-api-access-tb7mh\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.894456 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.894415    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.894656 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.894637    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr94q\" (UniqueName: \"kubernetes.io/projected/3a2a4927-992c-4f11-9f42-81405452f431-kube-api-access-dr94q\") pod \"kas-connection-checker-685895c7c9-x8bf8\" (UID: \"3a2a4927-992c-4f11-9f42-81405452f431\") " pod="kube-system/kas-connection-checker-685895c7c9-x8bf8"
Nov 02 21:57:43.895162 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.895141    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmgw\" (UniqueName: \"kubernetes.io/projected/c10347cb-927a-4d45-a48f-d9bc553d68db-kube-api-access-bbmgw\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:43.895602 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.895529    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.895891 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.895874    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qkb\" (UniqueName: \"kubernetes.io/projected/0530c38c-3055-48b0-a829-ad3068c97528-kube-api-access-z5qkb\") pod \"kas-connection-checker-685895c7c9-q9m4v\" (UID: \"0530c38c-3055-48b0-a829-ad3068c97528\") " pod="kube-system/kas-connection-checker-685895c7c9-q9m4v"
Nov 02 21:57:43.896220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.896201    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.896276 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.896220    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjs5\" (UniqueName: \"kubernetes.io/projected/6a19a824-6d01-4257-a6c5-90c2d7f0c649-kube-api-access-zzjs5\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.896460 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.896445    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhx6\" (UniqueName: \"kubernetes.io/projected/d7c34109-7853-45e8-a813-29b53a775844-kube-api-access-pjhx6\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.896737 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.896715    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/8d91b2f9-d641-449a-9bae-601f629cc281-kube-api-access-r98n2\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.897024 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897006    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcsq\" (UniqueName: \"kubernetes.io/projected/327d452e-cee7-493d-bd9a-81dc5b5b42e2-kube-api-access-xjcsq\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.897138 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897119    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7wv\" (UniqueName: \"kubernetes.io/projected/8031ebee-f5f9-4efa-acac-f56c132e93cf-kube-api-access-pn7wv\") pod \"network-check-target-kbg45\" (UID: \"8031ebee-f5f9-4efa-acac-f56c132e93cf\") " pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:43.897195 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897123    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpsh\" (UniqueName: \"kubernetes.io/projected/0a2571db-e8bb-431c-a74d-dea6f9a261cc-kube-api-access-xwpsh\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.897265 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897240    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhv2\" (UniqueName: \"kubernetes.io/projected/c61cf27a-32bb-4a64-9ef4-56badad04b6d-kube-api-access-dfhv2\") pod \"insights-operator-7c4986d449-jx4w4\" (UID: \"c61cf27a-32bb-4a64-9ef4-56badad04b6d\") " pod="openshift-insights/insights-operator-7c4986d449-jx4w4"
Nov 02 21:57:43.897784 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897766    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxsr\" (UniqueName: \"kubernetes.io/projected/e0f5aae5-63bc-4e64-a357-1b046c4fc470-kube-api-access-4lxsr\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.897845 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.897834    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhs5p\" (UniqueName: \"kubernetes.io/projected/ec070a08-538c-46a0-8f70-bf23a1468b0c-kube-api-access-bhs5p\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.921064 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921029    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 21:57:43.921186 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921084    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 21:57:43.921186 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921090    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 21:57:43.921186 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921107    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 21:57:43.921329 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921223    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 21:57:43.921329 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921229    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dx8bh\""
Nov 02 21:57:43.921329 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921246    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 21:57:43.921497 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921479    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7q497\""
Nov 02 21:57:43.921612 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921518    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 21:57:43.921612 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921589    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t5j6q\""
Nov 02 21:57:43.921717 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921664    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 21:57:43.921717 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921689    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 21:57:43.921717 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921689    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jdvwg\""
Nov 02 21:57:43.921806 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.921717    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 21:57:43.922571 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922555    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.922648 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922560    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 21:57:43.922648 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922601    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 21:57:43.922648 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922606    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 21:57:43.922648 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922626    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqhdf\""
Nov 02 21:57:43.922778 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922644    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g9f57\""
Nov 02 21:57:43.922778 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922688    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7c60948-0876-4c3c-822e-a954bf3f02fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hrts2\" (UID: \"f7c60948-0876-4c3c-822e-a954bf3f02fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2"
Nov 02 21:57:43.922839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922817    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec070a08-538c-46a0-8f70-bf23a1468b0c-serving-cert\") pod \"console-operator-7cd56b8dbc-jpvb6\" (UID: \"ec070a08-538c-46a0-8f70-bf23a1468b0c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:43.922839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922820    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 21:57:43.922904 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922872    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-447sr\""
Nov 02 21:57:43.922950 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922912    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 21:57:43.922950 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922915    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9dzjg\""
Nov 02 21:57:43.922950 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922933    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 21:57:43.922950 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922941    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sd22g\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922949    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xxf9z\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.922977    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923029    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vsl4h\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923032    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjf48\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923054    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923058    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6vj4v\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923067    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 21:57:43.923092 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923095    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6t4jd\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923161    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf8xp\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923166    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v8fd\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923198    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4qcf6\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923205    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtxk8\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923216    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923233    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n7kkn\""
Nov 02 21:57:43.923373 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923247    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tk4z\""
Nov 02 21:57:43.923661 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923650    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b8hrl\""
Nov 02 21:57:43.923722 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.923708    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f5aae5-63bc-4e64-a357-1b046c4fc470-ovn-node-metrics-cert\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.924481 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.924466    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/418c8848-c1f8-449c-9406-f6c70830cde5-serving-cert\") pod \"service-ca-operator-544d749c8c-6hbtm\" (UID: \"418c8848-c1f8-449c-9406-f6c70830cde5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm"
Nov 02 21:57:43.924526 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.924489    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.924649 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.924636    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-default-certificate\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.926924 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.926783    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.926924 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.926894    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.926924 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.926916    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c10347cb-927a-4d45-a48f-d9bc553d68db-cert\") pod \"ingress-canary-lknc7\" (UID: \"c10347cb-927a-4d45-a48f-d9bc553d68db\") " pod="openshift-ingress-canary/ingress-canary-lknc7"
Nov 02 21:57:43.927265 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.927000    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36174814-afd3-42b6-a555-75c832ebacf5-agent-certs\") pod \"konnectivity-agent-kzxpt\" (UID: \"36174814-afd3-42b6-a555-75c832ebacf5\") " pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:43.927321 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.927283    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-stats-auth\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.932460 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.932440    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d39c020-6a26-47a2-8f57-1655acdc05f0-original-pull-secret\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.987571 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987519    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.987571 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987569    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987588    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987610    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987633    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987636    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-ovn\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987646    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2afa6f34-35a6-4bcd-939e-8083b7cb4fb3-host\") pod \"node-ca-85npx\" (UID: \"2afa6f34-35a6-4bcd-939e-8083b7cb4fb3\") " pod="openshift-image-registry/node-ca-85npx"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987653    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987693    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987700    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-cnibin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987708    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987721    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987736    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987739    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-device-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987757    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-etc-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987745    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.987783 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987789    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987804    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987821    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.987822    4929 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987849    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987869    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-kubelet\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987874    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987893    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-lib-modules\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987891    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-cnibin\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.987923    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.487901552 +0000 UTC m=+1.795992354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : secret "metrics-daemon-secret" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987928    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-slash\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.987942    4929 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987969    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988003    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.487992972 +0000 UTC m=+1.796083778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.987944    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-conf-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988037    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988064    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988082    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988098    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988119    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988135    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988151    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988160    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-kubelet-config\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988171    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988175    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-systemd-units\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988152    4929 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988222    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/730c3bbf-87e5-4378-a258-a990d610143c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988227    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-var-lib-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988239    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-sys-fs\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988245    4929 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988266    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls podName:d7c34109-7853-45e8-a813-29b53a775844 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488247336 +0000 UTC m=+1.796338126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sv8qs" (UID: "d7c34109-7853-45e8-a813-29b53a775844") : secret "samples-operator-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988195    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988268    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988301    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488291276 +0000 UTC m=+1.796382059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988308    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988318    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988342    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988357    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-system-cni-dir\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988359    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988186    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-systemd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988387    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488375221 +0000 UTC m=+1.796466023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988391    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-hostroot\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988423    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988446    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488435959 +0000 UTC m=+1.796526781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988467    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988485    4929 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:57:43.988445 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988489    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988512    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988520    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls podName:feae0ada-a336-48c6-a163-7a968597ad33 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488509026 +0000 UTC m=+1.796599814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls") pod "dns-default-kcsw9" (UID: "feae0ada-a336-48c6-a163-7a968597ad33") : secret "dns-default-metrics-tls" not found
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988519    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988479    4929 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988553    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a2571db-e8bb-431c-a74d-dea6f9a261cc-host-slash\") pod \"iptables-alerter-6rjz9\" (UID: \"0a2571db-e8bb-431c-a74d-dea6f9a261cc\") " pod="openshift-network-operator/iptables-alerter-6rjz9"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988559    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988574    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-socket-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988601    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-system-cni-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988615    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988640    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988667    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988691    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988712    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988728    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988738    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-netns\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988734    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988768    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-etc-kubernetes\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988732    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988790    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988788    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d39c020-6a26-47a2-8f57-1655acdc05f0-dbus\") pod \"global-pull-secret-syncer-bvq2f\" (UID: \"9d39c020-6a26-47a2-8f57-1655acdc05f0\") " pod="kube-system/global-pull-secret-syncer-bvq2f"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988793    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-k8s-cni-cncf-io\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988776    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988803    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-kubelet\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.988821    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.488808885 +0000 UTC m=+1.796899683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988853    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988881    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988897    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988904    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-netd\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988918    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988943    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-node-log\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988943    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988945    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988977    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-bin\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988984    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.988984    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/730c3bbf-87e5-4378-a258-a990d610143c-crio-socket\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989020    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989046    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989051    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-systemd\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989077    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989100    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989101    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f05bd7-b4eb-4df9-af57-fb0277839916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vxz77\" (UID: \"b2f05bd7-b4eb-4df9-af57-fb0277839916\") " pod="openshift-multus/multus-additional-cni-plugins-vxz77"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989122    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989147    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989150    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d91b2f9-d641-449a-9bae-601f629cc281-registration-dir\") pod \"aws-ebs-csi-driver-node-bsgcv\" (UID: \"8d91b2f9-d641-449a-9bae-601f629cc281\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsgcv"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989154    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysconfig\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989168    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989178    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-host\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989190    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989216    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989225    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-run\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989244    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989277    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989300    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989303    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-var-lib-kubelet\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989329    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989368    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989375    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989394    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989412    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-run-openvswitch\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989394    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-kubernetes\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989433    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf18c217-668e-4f22-8bb3-ea403e33e962-hosts-file\") pod \"node-resolver-xcrch\" (UID: \"bf18c217-668e-4f22-8bb3-ea403e33e962\") " pod="openshift-dns/node-resolver-xcrch"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989453    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989488    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989507    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-run-netns\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989554    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-modprobe-d\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989600    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-etc-sysctl-conf\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989608    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/342f55ac-11b3-442a-beb6-f2aadf8aa899-sys\") pod \"tuned-r9brh\" (UID: \"342f55ac-11b3-442a-beb6-f2aadf8aa899\") " pod="openshift-cluster-node-tuning-operator/tuned-r9brh"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989657    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-host-cni-bin\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989692    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0f5aae5-63bc-4e64-a357-1b046c4fc470-log-socket\") pod \"ovnkube-node-55qz7\" (UID: \"e0f5aae5-63bc-4e64-a357-1b046c4fc470\") " pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989736    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-multus-socket-dir-parent\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989743    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989775    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989800    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-run-multus-certs\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989780    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a19a824-6d01-4257-a6c5-90c2d7f0c649-host-var-lib-cni-multus\") pod \"multus-hclsq\" (UID: \"6a19a824-6d01-4257-a6c5-90c2d7f0c649\") " pod="openshift-multus/multus-hclsq"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.989806    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.989877    4929 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:43.990202 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:43.989932    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:44.489917411 +0000 UTC m=+1.798008208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:43.992624 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:43.991511    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327d452e-cee7-493d-bd9a-81dc5b5b42e2-metrics-certs\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:44.119280 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.119248    4929 scope.go:122] "RemoveContainer" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"
Nov 02 21:57:44.494672 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494581    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:44.494672 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494638    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:44.494672 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494669    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494701    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494747    4929 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494798    4929 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494821    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.494806489 +0000 UTC m=+2.802897284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494842    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494848    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.494836201 +0000 UTC m=+2.802926988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494855    4929 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494873    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494915    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.494903316 +0000 UTC m=+2.802994101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : secret "metrics-daemon-secret" not found
Nov 02 21:57:44.494944 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494929    4929 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494955    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494967    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494974    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls podName:730c3bbf-87e5-4378-a258-a990d610143c nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.494960406 +0000 UTC m=+2.803051193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdmzs" (UID: "730c3bbf-87e5-4378-a258-a990d610143c") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.494932    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.494994    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.49498482 +0000 UTC m=+2.803075604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.495011    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.495004706 +0000 UTC m=+2.803095490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.495033    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.495063    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.495156    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.495163    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:44.495322 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.495188    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:45.495181385 +0000 UTC m=+2.803272168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:44.497140 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.497117    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7c34109-7853-45e8-a813-29b53a775844-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sv8qs\" (UID: \"d7c34109-7853-45e8-a813-29b53a775844\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:44.497234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.497164    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/feae0ada-a336-48c6-a163-7a968597ad33-metrics-tls\") pod \"dns-default-kcsw9\" (UID: \"feae0ada-a336-48c6-a163-7a968597ad33\") " pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:44.718201 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.718141    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"
Nov 02 21:57:44.731669 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.731640    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:44.856107 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.856076    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 21:57:44.856584 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.856568    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/1.log"
Nov 02 21:57:44.856646 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.856600    4929 generic.go:356] "Generic (PLEG): container finished" podID="ec070a08-538c-46a0-8f70-bf23a1468b0c" containerID="9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9" exitCode=255
Nov 02 21:57:44.856678 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.856645    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" event={"ID":"ec070a08-538c-46a0-8f70-bf23a1468b0c","Type":"ContainerDied","Data":"9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9"}
Nov 02 21:57:44.856718 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.856706    4929 scope.go:122] "RemoveContainer" containerID="89a7969708aa430e06100cdf816845f9e8b2877870e4dc2d5b8d19d14c840888"
Nov 02 21:57:44.857148 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:44.857129    4929 scope.go:122] "RemoveContainer" containerID="9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9"
Nov 02 21:57:44.857311 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:44.857288    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:45.017079 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.017016    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs"]
Nov 02 21:57:45.049663 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.049419    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kcsw9"]
Nov 02 21:57:45.052593 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:45.052561    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeae0ada_a336_48c6_a163_7a968597ad33.slice/crio-a3243aa164d9611d534ec3406669624bc0b5a49d3de45ac63ec5b1ba1e0dac9d WatchSource:0}: Error finding container a3243aa164d9611d534ec3406669624bc0b5a49d3de45ac63ec5b1ba1e0dac9d: Status 404 returned error can't find the container with id a3243aa164d9611d534ec3406669624bc0b5a49d3de45ac63ec5b1ba1e0dac9d
Nov 02 21:57:45.054674 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.054657    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:57:45.505571 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505501    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505596    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505623    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505644    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505661    4929 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505774    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.505755264 +0000 UTC m=+4.813846063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505775    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505794    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505795    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:45.505839 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505844    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.505828979 +0000 UTC m=+4.813919776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505872    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505881    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505871    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.505861144 +0000 UTC m=+4.813951940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.505940    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.50591989 +0000 UTC m=+4.814010678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.505991    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.506024    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.506108    4929 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.506112    4929 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.506155    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.506143369 +0000 UTC m=+4.814234174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:45.506237 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:45.506170    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:47.506163934 +0000 UTC m=+4.814254744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : secret "metrics-daemon-secret" not found
Nov 02 21:57:45.507949 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.507925    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/730c3bbf-87e5-4378-a258-a990d610143c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdmzs\" (UID: \"730c3bbf-87e5-4378-a258-a990d610143c\") " pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:45.574334 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.574307    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:45.631878 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.631844    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdmzs"
Nov 02 21:57:45.864301 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.864266    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcsw9" event={"ID":"feae0ada-a336-48c6-a163-7a968597ad33","Type":"ContainerStarted","Data":"a3243aa164d9611d534ec3406669624bc0b5a49d3de45ac63ec5b1ba1e0dac9d"}
Nov 02 21:57:45.866336 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.866306    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs" event={"ID":"d7c34109-7853-45e8-a813-29b53a775844","Type":"ContainerStarted","Data":"3dda7ca956f99064c5692b301ebb598874a0aa5176a01fdb9b5c0f47936bb7a2"}
Nov 02 21:57:45.868230 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.868204    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 21:57:45.920380 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:45.920339    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdmzs"]
Nov 02 21:57:45.924952 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:45.924920    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730c3bbf_87e5_4378_a258_a990d610143c.slice/crio-94b924f77d48db62eae8a840cac12081d809564e8aceb7c32c8c5093a0c028d6 WatchSource:0}: Error finding container 94b924f77d48db62eae8a840cac12081d809564e8aceb7c32c8c5093a0c028d6: Status 404 returned error can't find the container with id 94b924f77d48db62eae8a840cac12081d809564e8aceb7c32c8c5093a0c028d6
Nov 02 21:57:46.062234 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:46.062204    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcrch_bf18c217-668e-4f22-8bb3-ea403e33e962/dns-node-resolver/0.log"
Nov 02 21:57:46.662716 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:46.662682    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85npx_2afa6f34-35a6-4bcd-939e-8083b7cb4fb3/node-ca/0.log"
Nov 02 21:57:46.873546 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:46.873492    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdmzs" event={"ID":"730c3bbf-87e5-4378-a258-a990d610143c","Type":"ContainerStarted","Data":"b1baea1d3c860d978e9c42e9c122638837a0de3cd16a1d12af9519de1955730b"}
Nov 02 21:57:46.873888 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:46.873575    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdmzs" event={"ID":"730c3bbf-87e5-4378-a258-a990d610143c","Type":"ContainerStarted","Data":"d597bc311342785982527cb8ee5d03fa4bf81fc7b81d9950e2bdad4a798f72a8"}
Nov 02 21:57:46.873888 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:46.873596    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdmzs" event={"ID":"730c3bbf-87e5-4378-a258-a990d610143c","Type":"ContainerStarted","Data":"94b924f77d48db62eae8a840cac12081d809564e8aceb7c32c8c5093a0c028d6"}
Nov 02 21:57:47.526387 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526345    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:47.526387 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526391    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526415    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526457    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526482    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.526501    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526524    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526564    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-998796474-kqqtd: secret "image-registry-tls" not found
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526573    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526524912 +0000 UTC m=+8.834615712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526617    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls podName:e87a2020-99e4-4d23-bd7c-47ed49b6925e nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526600362 +0000 UTC m=+8.834691163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls") pod "image-registry-998796474-kqqtd" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e") : secret "image-registry-tls" not found
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526638    4929 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:47.526662 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526654    4929 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526675    4929 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57bf9d85b7-pxw8s: secret "image-registry-tls" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526691    4929 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526700    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526685079 +0000 UTC m=+8.834775880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526642    4929 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526727    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs podName:b911671e-958f-441e-92cc-bf023d0757a7 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526714902 +0000 UTC m=+8.834805702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs") pod "network-metrics-daemon-nvvxb" (UID: "b911671e-958f-441e-92cc-bf023d0757a7") : secret "metrics-daemon-secret" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526742    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls podName:a113cb77-4c64-47ea-8d09-723c2146038f nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526732648 +0000 UTC m=+8.834823430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g5w5w" (UID: "a113cb77-4c64-47ea-8d09-723c2146038f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 21:57:47.527088 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.526765    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls podName:cd31455a-0dba-4b59-b254-39739cbe0790 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:51.526747882 +0000 UTC m=+8.834838679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls") pod "image-registry-57bf9d85b7-pxw8s" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790") : secret "image-registry-tls" not found
Nov 02 21:57:47.662912 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.662884    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lknc7_c10347cb-927a-4d45-a48f-d9bc553d68db/serve-healthcheck-canary/0.log"
Nov 02 21:57:47.712069 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.712029    4929 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:47.712529 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.712506    4929 scope.go:122] "RemoveContainer" containerID="9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9"
Nov 02 21:57:47.712759 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:47.712736    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:47.879233 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.879194    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcsw9" event={"ID":"feae0ada-a336-48c6-a163-7a968597ad33","Type":"ContainerStarted","Data":"ff6306ecc89ecfc7603e462f17a8e16d31f4ff38f42d5f2e0dc5dfef91fb7e32"}
Nov 02 21:57:47.879724 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.879259    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcsw9" event={"ID":"feae0ada-a336-48c6-a163-7a968597ad33","Type":"ContainerStarted","Data":"f692991a1a766cd87bcf35beff43abef364ca6e524e5109988ea0568f8a1c5bc"}
Nov 02 21:57:47.879724 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.879288    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:47.881087 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.881062    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs" event={"ID":"d7c34109-7853-45e8-a813-29b53a775844","Type":"ContainerStarted","Data":"21046c1d8ff042ef7949a96425cd3fbc177414d86ae4ad568434fff4d734880a"}
Nov 02 21:57:47.881201 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:47.881103    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sv8qs" event={"ID":"d7c34109-7853-45e8-a813-29b53a775844","Type":"ContainerStarted","Data":"2266a17e56a5cf46bbdeb7ead9ad0845537f16c1011b12a2681229ad616234e0"}
Nov 02 21:57:48.062425 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:48.062396    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xbm66_786d8c8c-f90c-468b-8833-b51c7e4965b8/migrator/0.log"
Nov 02 21:57:48.263944 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:48.263868    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hrts2_f7c60948-0876-4c3c-822e-a954bf3f02fb/kube-storage-version-migrator-operator/0.log"
Nov 02 21:57:48.765033 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:48.764987    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:48.765390 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:48.765376    4929 scope.go:122] "RemoveContainer" containerID="9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9"
Nov 02 21:57:48.765609 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:48.765589    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jpvb6_openshift-console-operator(ec070a08-538c-46a0-8f70-bf23a1468b0c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" podUID="ec070a08-538c-46a0-8f70-bf23a1468b0c"
Nov 02 21:57:48.886990 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:48.886901    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdmzs" event={"ID":"730c3bbf-87e5-4378-a258-a990d610143c","Type":"ContainerStarted","Data":"cd886f338dd963644958c512a967b25c92db2973fc2de76af81d29133230ffd0"}
Nov 02 21:57:51.469295 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.469265    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:57:51.469686 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.469457    4929 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:51.563518 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563471    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:51.563713 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563557    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:51.563713 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563592    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:51.563713 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563627    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:51.563841 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563710    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:51.563841 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:51.563780    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle podName:327d452e-cee7-493d-bd9a-81dc5b5b42e2 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:59.563756601 +0000 UTC m=+16.871847411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle") pod "router-default-8469b8f4f5-bf9r8" (UID: "327d452e-cee7-493d-bd9a-81dc5b5b42e2") : configmap references non-existent config key: service-ca.crt
Nov 02 21:57:51.563841 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.563831    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:51.564132 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:51.564107    4929 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 21:57:51.564227 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:51.564175    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert podName:7d36c5e6-09a7-4301-b6a1-db669e212d61 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:59.564157042 +0000 UTC m=+16.872247824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vnfgs" (UID: "7d36c5e6-09a7-4301-b6a1-db669e212d61") : secret "networking-console-plugin-cert" not found
Nov 02 21:57:51.566109 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.566086    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911671e-958f-441e-92cc-bf023d0757a7-metrics-certs\") pod \"network-metrics-daemon-nvvxb\" (UID: \"b911671e-958f-441e-92cc-bf023d0757a7\") " pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:51.566377 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.566354    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"image-registry-998796474-kqqtd\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") " pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:51.566475 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.566457    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a113cb77-4c64-47ea-8d09-723c2146038f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g5w5w\" (UID: \"a113cb77-4c64-47ea-8d09-723c2146038f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:51.566511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.566481    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"image-registry-57bf9d85b7-pxw8s\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") " pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:51.619015 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.618985    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"
Nov 02 21:57:51.621156 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.621137    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:51.622213 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.622197    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvvxb"
Nov 02 21:57:51.630396 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.630372    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:57:51.905928 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.905897    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w"]
Nov 02 21:57:51.911426 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.911401    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:57:51.916095 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:51.916069    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87a2020_99e4_4d23_bd7c_47ed49b6925e.slice/crio-462d1962c4acb9aad1c3cbdcdfea4aac03c6880ec4004a02cf2b588a0aeeb699 WatchSource:0}: Error finding container 462d1962c4acb9aad1c3cbdcdfea4aac03c6880ec4004a02cf2b588a0aeeb699: Status 404 returned error can't find the container with id 462d1962c4acb9aad1c3cbdcdfea4aac03c6880ec4004a02cf2b588a0aeeb699
Nov 02 21:57:51.931359 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.931334    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:57:51.934382 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:51.934361    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nvvxb"]
Nov 02 21:57:51.942748 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:57:51.942727    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb911671e_958f_441e_92cc_bf023d0757a7.slice/crio-51b248f117ed0881913936c62d75943a6d0473236861cb47c75384eaf1ad981a WatchSource:0}: Error finding container 51b248f117ed0881913936c62d75943a6d0473236861cb47c75384eaf1ad981a: Status 404 returned error can't find the container with id 51b248f117ed0881913936c62d75943a6d0473236861cb47c75384eaf1ad981a
Nov 02 21:57:52.906294 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.904597    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvvxb" event={"ID":"b911671e-958f-441e-92cc-bf023d0757a7","Type":"ContainerStarted","Data":"51b248f117ed0881913936c62d75943a6d0473236861cb47c75384eaf1ad981a"}
Nov 02 21:57:52.907219 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.907121    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-998796474-kqqtd" event={"ID":"e87a2020-99e4-4d23-bd7c-47ed49b6925e","Type":"ContainerStarted","Data":"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"}
Nov 02 21:57:52.907340 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.907245    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-998796474-kqqtd" event={"ID":"e87a2020-99e4-4d23-bd7c-47ed49b6925e","Type":"ContainerStarted","Data":"462d1962c4acb9aad1c3cbdcdfea4aac03c6880ec4004a02cf2b588a0aeeb699"}
Nov 02 21:57:52.909237 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.909203    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s" event={"ID":"cd31455a-0dba-4b59-b254-39739cbe0790","Type":"ContainerStarted","Data":"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"}
Nov 02 21:57:52.909350 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.909262    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s" event={"ID":"cd31455a-0dba-4b59-b254-39739cbe0790","Type":"ContainerStarted","Data":"58772a800a955dde355dc4db51d03c0810ee5bad482085af290f244f2c5d7e21"}
Nov 02 21:57:52.910266 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:52.910242    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w" event={"ID":"a113cb77-4c64-47ea-8d09-723c2146038f","Type":"ContainerStarted","Data":"c951fa011d5949396fe956bd383f7bec8c38a5637ef57657928db7f88cb73ad4"}
Nov 02 21:57:53.617882 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:53.617834    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:57:53.916009 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:53.915926    4929 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:57:54.060990 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:54.060949    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:57:54.801998 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:54.801969    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kzxpt"
Nov 02 21:57:54.922020 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:54.921989    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvvxb" event={"ID":"b911671e-958f-441e-92cc-bf023d0757a7","Type":"ContainerStarted","Data":"a77c3a763603a5fa4cdc85adfa33bf09eeb0f4c5cd5645cff1e8e0ce4c417ccb"}
Nov 02 21:57:54.922359 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:54.922045    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvvxb" event={"ID":"b911671e-958f-441e-92cc-bf023d0757a7","Type":"ContainerStarted","Data":"5f971c815795ee2cad0c2db8a976f3c7d3ce74bdc7529664af79725f25e0dae2"}
Nov 02 21:57:55.797725 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:55.797689    4929 scope.go:122] "RemoveContainer" containerID="9599326dfda9cea9bc174fb88a62b09a6cbc3a660e0d982aa62a50660da283b9"
Nov 02 21:57:55.927317 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:55.927275    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g5w5w" event={"ID":"a113cb77-4c64-47ea-8d09-723c2146038f","Type":"ContainerStarted","Data":"e3ca0a239632213766d2d599803a1d1da77c77ccb4f7a99c688606b605dae9b5"}
Nov 02 21:57:55.929219 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:55.929200    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 21:57:55.929327 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:55.929308    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6" event={"ID":"ec070a08-538c-46a0-8f70-bf23a1468b0c","Type":"ContainerStarted","Data":"b5d97e2be4e9dd5df56115c1214f17050d944f45638a73cea603ca86f12b55dc"}
Nov 02 21:57:55.929689 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:55.929671    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:56.190688 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:56.190659    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jpvb6"
Nov 02 21:57:57.495203 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:57:57.495167    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 21:57:57.888895 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:57.888866    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kcsw9"
Nov 02 21:57:58.572131 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:58.572097    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:57:59.639362 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.639322    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:59.639872 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.639395    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:59.640119 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.640094    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327d452e-cee7-493d-bd9a-81dc5b5b42e2-service-ca-bundle\") pod \"router-default-8469b8f4f5-bf9r8\" (UID: \"327d452e-cee7-493d-bd9a-81dc5b5b42e2\") " pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:59.641748 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.641726    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d36c5e6-09a7-4301-b6a1-db669e212d61-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vnfgs\" (UID: \"7d36c5e6-09a7-4301-b6a1-db669e212d61\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:57:59.719784 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.719748    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:57:59.720801 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:57:59.720782    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"
Nov 02 21:58:00.000765 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.000694    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vnfgs"]
Nov 02 21:58:00.003668 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:58:00.003625    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d36c5e6_09a7_4301_b6a1_db669e212d61.slice/crio-57ef93303501c4edfb8a33e5c5ce63305187be31a20c79abffdcc16aad805289 WatchSource:0}: Error finding container 57ef93303501c4edfb8a33e5c5ce63305187be31a20c79abffdcc16aad805289: Status 404 returned error can't find the container with id 57ef93303501c4edfb8a33e5c5ce63305187be31a20c79abffdcc16aad805289
Nov 02 21:58:00.005648 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.005620    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8469b8f4f5-bf9r8"]
Nov 02 21:58:00.008863 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:58:00.008830    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327d452e_cee7_493d_bd9a_81dc5b5b42e2.slice/crio-1607d9d24ca5cddac054de310ec506614a6740156a8f302f40f353b699440107 WatchSource:0}: Error finding container 1607d9d24ca5cddac054de310ec506614a6740156a8f302f40f353b699440107: Status 404 returned error can't find the container with id 1607d9d24ca5cddac054de310ec506614a6740156a8f302f40f353b699440107
Nov 02 21:58:00.570364 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.570249    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:58:00.946428 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.946385    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs" event={"ID":"7d36c5e6-09a7-4301-b6a1-db669e212d61","Type":"ContainerStarted","Data":"57ef93303501c4edfb8a33e5c5ce63305187be31a20c79abffdcc16aad805289"}
Nov 02 21:58:00.948247 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.948217    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8" event={"ID":"327d452e-cee7-493d-bd9a-81dc5b5b42e2","Type":"ContainerStarted","Data":"3f610af030f5756343aa0355abba254d723664b1c278ca755b984d8d949996ba"}
Nov 02 21:58:00.948379 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:00.948267    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8" event={"ID":"327d452e-cee7-493d-bd9a-81dc5b5b42e2","Type":"ContainerStarted","Data":"1607d9d24ca5cddac054de310ec506614a6740156a8f302f40f353b699440107"}
Nov 02 21:58:01.720295 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:01.720251    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:58:01.723059 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:01.723035    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:58:01.954301 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:01.954261    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vnfgs" event={"ID":"7d36c5e6-09a7-4301-b6a1-db669e212d61","Type":"ContainerStarted","Data":"d3f888afd1c49a009e85d70700a062eb444d7545e59ea3973a52b6231e48f91f"}
Nov 02 21:58:01.954818 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:01.954505    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:58:01.955810 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:01.955791    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8469b8f4f5-bf9r8"
Nov 02 21:58:02.310161 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:02.310073    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55qz7"
Nov 02 21:58:08.577391 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:08.577357    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:58:10.575391 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:10.575358    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:58:11.556529 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.556494    4929 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log" limit=1024
Nov 02 21:58:11.654513 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.654483    4929 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log" limit=1024
Nov 02 21:58:11.856521 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.856475    4929 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log" limit=1024
Nov 02 21:58:11.857217 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.857194    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r2cqw"]
Nov 02 21:58:11.893102 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.893070    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.896615 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.896580    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 21:58:11.896615 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.896567    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 21:58:11.896790 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.896651    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xgmtc\""
Nov 02 21:58:11.927894 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.927854    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 21:58:11.928049 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.927905    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 21:58:11.941170 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.941136    4929 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log" limit=1024
Nov 02 21:58:11.957191 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957162    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khd8\" (UniqueName: \"kubernetes.io/projected/c7914af0-6869-48fb-9b29-38c2da5b9dcd-kube-api-access-2khd8\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957356 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957199    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-tls\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957356 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957226    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-sys\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957356 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957300    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-wtmp\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957356 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957333    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957404    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957430    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-root\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957511 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957449    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-textfile\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:11.957638 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:11.957512    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-metrics-client-ca\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.043586 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.043529    4929 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log" limit=1024
Nov 02 21:58:12.058268 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058240    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058386 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058274    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-root\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058386 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058293    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-textfile\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058386 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058318    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-metrics-client-ca\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058386 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058360    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-root\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058386 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058375    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khd8\" (UniqueName: \"kubernetes.io/projected/c7914af0-6869-48fb-9b29-38c2da5b9dcd-kube-api-access-2khd8\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058619 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058416    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-tls\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058619 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058458    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-sys\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058619 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058482    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-wtmp\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.058960 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058878    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-sys\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.059145 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058974    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-textfile\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.059265 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.058505    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.059415 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.059095    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-wtmp\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.059522 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.059434    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-metrics-client-ca\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.059650 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.059624    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.061662 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.061640    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.065447 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.065426    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7914af0-6869-48fb-9b29-38c2da5b9dcd-node-exporter-tls\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.067004 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.066977    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khd8\" (UniqueName: \"kubernetes.io/projected/c7914af0-6869-48fb-9b29-38c2da5b9dcd-kube-api-access-2khd8\") pod \"node-exporter-r2cqw\" (UID: \"c7914af0-6869-48fb-9b29-38c2da5b9dcd\") " pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.243792 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.243709    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2cqw"
Nov 02 21:58:12.288074 ip-10-0-131-57 kubenswrapper[4929]: W1102 21:58:12.288046    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7914af0_6869_48fb_9b29_38c2da5b9dcd.slice/crio-70e89190bf46c2ec3c26f9c2b8c2c361011c2cd17e8afe39144c10fc40de0307 WatchSource:0}: Error finding container 70e89190bf46c2ec3c26f9c2b8c2c361011c2cd17e8afe39144c10fc40de0307: Status 404 returned error can't find the container with id 70e89190bf46c2ec3c26f9c2b8c2c361011c2cd17e8afe39144c10fc40de0307
Nov 02 21:58:12.996515 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:12.996479    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2cqw" event={"ID":"c7914af0-6869-48fb-9b29-38c2da5b9dcd","Type":"ContainerStarted","Data":"70e89190bf46c2ec3c26f9c2b8c2c361011c2cd17e8afe39144c10fc40de0307"}
Nov 02 21:58:14.064149 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:14.064118    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kbg45"
Nov 02 21:58:15.006130 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:15.006094    4929 generic.go:356] "Generic (PLEG): container finished" podID="c7914af0-6869-48fb-9b29-38c2da5b9dcd" containerID="04c3c413c18252893c496fcfd131eed77b151b55b89c064f729808049b269836" exitCode=0
Nov 02 21:58:15.006307 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:15.006178    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2cqw" event={"ID":"c7914af0-6869-48fb-9b29-38c2da5b9dcd","Type":"ContainerDied","Data":"04c3c413c18252893c496fcfd131eed77b151b55b89c064f729808049b269836"}
Nov 02 21:58:16.012632 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:16.012591    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2cqw" event={"ID":"c7914af0-6869-48fb-9b29-38c2da5b9dcd","Type":"ContainerStarted","Data":"495147fe461f55b12fb196a54c4cd5a0aaf20fef42232228b9f43836a961d360"}
Nov 02 21:58:16.013022 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:16.012692    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2cqw" event={"ID":"c7914af0-6869-48fb-9b29-38c2da5b9dcd","Type":"ContainerStarted","Data":"e5aceec1380d8f4228c17899c7e4f45702f57e8874f8c17ea9f7fae0723ec3f3"}
Nov 02 21:58:16.039646 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:16.039572    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r2cqw" podStartSLOduration=2.81003502 podStartE2EDuration="5.03955764s" totalImagesPullingTime="2.14933641s" totalInitContainerRuntime="80.18621ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:16.039080811 +0000 UTC m=+33.347171615" watchObservedRunningTime="2026-11-02 21:58:16.03955764 +0000 UTC m=+33.347648463"
Nov 02 21:58:16.484510 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:16.484473    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:58:23.598317 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.598276    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-998796474-kqqtd" podUID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" containerName="registry" containerID="cri-o://1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082" gracePeriod=30
Nov 02 21:58:23.848675 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.848619    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:58:23.961674 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961638    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.961871 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961705    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.961871 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961727    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.961871 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961745    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.961871 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961861    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.962090 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961911    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.962090 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.961951    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.962090 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.962007    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") pod \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\" (UID: \"e87a2020-99e4-4d23-bd7c-47ed49b6925e\") "
Nov 02 21:58:23.962244 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.962205    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:58:23.962295 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.962244    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:58:23.964722 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.964686    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:23.964860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.964695    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:23.964860 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.964808    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "kube-api-access-nhglq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:23.964998 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.964975    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:58:23.965056 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.965032    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:58:23.968900 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:23.968879    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted" pod "e87a2020-99e4-4d23-bd7c-47ed49b6925e" (UID: "e87a2020-99e4-4d23-bd7c-47ed49b6925e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:58:24.045061 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.045026    4929 generic.go:356] "Generic (PLEG): container finished" podID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" containerID="1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082" exitCode=0
Nov 02 21:58:24.045220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.045069    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-998796474-kqqtd"
Nov 02 21:58:24.045220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.045104    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-998796474-kqqtd" event={"ID":"e87a2020-99e4-4d23-bd7c-47ed49b6925e","Type":"ContainerDied","Data":"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"}
Nov 02 21:58:24.045220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.045157    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-998796474-kqqtd" event={"ID":"e87a2020-99e4-4d23-bd7c-47ed49b6925e","Type":"ContainerDied","Data":"462d1962c4acb9aad1c3cbdcdfea4aac03c6880ec4004a02cf2b588a0aeeb699"}
Nov 02 21:58:24.045220 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.045178    4929 scope.go:122] "RemoveContainer" containerID="1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"
Nov 02 21:58:24.053898 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.053876    4929 scope.go:122] "RemoveContainer" containerID="1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"
Nov 02 21:58:24.054150 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:58:24.054131    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082\": container with ID starting with 1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082 not found: ID does not exist" containerID="1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"
Nov 02 21:58:24.054206 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.054159    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082"} err="failed to get container status \"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082\": rpc error: code = NotFound desc = could not find container \"1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082\": container with ID starting with 1c348cd44fdfa3b976ee98a12bb90ad2bbaf5702b98788282502b4542588e082 not found: ID does not exist"
Nov 02 21:58:24.062707 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062685    4929 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062707 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062706    4929 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-bound-sa-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062718    4929 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-trusted-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062728    4929 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e87a2020-99e4-4d23-bd7c-47ed49b6925e-registry-certificates\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062738    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhglq\" (UniqueName: \"kubernetes.io/projected/e87a2020-99e4-4d23-bd7c-47ed49b6925e-kube-api-access-nhglq\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062747    4929 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e87a2020-99e4-4d23-bd7c-47ed49b6925e-ca-trust-extracted\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062792    4929 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-installation-pull-secrets\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.062836 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.062807    4929 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e87a2020-99e4-4d23-bd7c-47ed49b6925e-image-registry-private-configuration\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:24.064422 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.064400    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:58:24.068003 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.067985    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-998796474-kqqtd"]
Nov 02 21:58:24.802177 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:24.802105    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" path="/var/lib/kubelet/pods/e87a2020-99e4-4d23-bd7c-47ed49b6925e/volumes"
Nov 02 21:58:27.496598 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:58:27.496557    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 21:58:41.503904 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.503846    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s" podUID="cd31455a-0dba-4b59-b254-39739cbe0790" containerName="registry" containerID="cri-o://c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e" gracePeriod=30
Nov 02 21:58:41.756413 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.756352    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:58:41.907465 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907430    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907465 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907468    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907487    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907525    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907595    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907641    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907669    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.907734 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907712    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") pod \"cd31455a-0dba-4b59-b254-39739cbe0790\" (UID: \"cd31455a-0dba-4b59-b254-39739cbe0790\") "
Nov 02 21:58:41.908094 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.907985    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:58:41.908094 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.908028    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:58:41.909959 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.909932    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:41.910125 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.910095    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:58:41.910354 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.910336    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:41.910460 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.910429    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:58:41.910577 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.910562    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "kube-api-access-gwp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:58:41.914832 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:41.914804    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted" pod "cd31455a-0dba-4b59-b254-39739cbe0790" (UID: "cd31455a-0dba-4b59-b254-39739cbe0790"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008884    4929 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-bound-sa-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008930    4929 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-registry-certificates\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008945    4929 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd31455a-0dba-4b59-b254-39739cbe0790-trusted-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008955    4929 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd31455a-0dba-4b59-b254-39739cbe0790-ca-trust-extracted\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008966    4929 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-installation-pull-secrets\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008977    4929 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd31455a-0dba-4b59-b254-39739cbe0790-image-registry-private-configuration\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008987    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwp6z\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-kube-api-access-gwp6z\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.008980 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.008996    4929 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd31455a-0dba-4b59-b254-39739cbe0790-registry-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 21:58:42.115986 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.115952    4929 generic.go:356] "Generic (PLEG): container finished" podID="cd31455a-0dba-4b59-b254-39739cbe0790" containerID="c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e" exitCode=0
Nov 02 21:58:42.115986 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.115990    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s" event={"ID":"cd31455a-0dba-4b59-b254-39739cbe0790","Type":"ContainerDied","Data":"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"}
Nov 02 21:58:42.116233 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.115999    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s"
Nov 02 21:58:42.116233 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.116036    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57bf9d85b7-pxw8s" event={"ID":"cd31455a-0dba-4b59-b254-39739cbe0790","Type":"ContainerDied","Data":"58772a800a955dde355dc4db51d03c0810ee5bad482085af290f244f2c5d7e21"}
Nov 02 21:58:42.116233 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.116058    4929 scope.go:122] "RemoveContainer" containerID="c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"
Nov 02 21:58:42.124420 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.124401    4929 scope.go:122] "RemoveContainer" containerID="c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"
Nov 02 21:58:42.124722 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:58:42.124704    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e\": container with ID starting with c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e not found: ID does not exist" containerID="c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"
Nov 02 21:58:42.124806 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.124731    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e"} err="failed to get container status \"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e\": rpc error: code = NotFound desc = could not find container \"c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e\": container with ID starting with c72ae6d1f801dda93036c5bbba6c0dd589e863c51c513fe4d85edf52b1c7229e not found: ID does not exist"
Nov 02 21:58:42.137402 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.137370    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:58:42.141339 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.141310    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57bf9d85b7-pxw8s"]
Nov 02 21:58:42.801960 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:42.801927    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd31455a-0dba-4b59-b254-39739cbe0790" path="/var/lib/kubelet/pods/cd31455a-0dba-4b59-b254-39739cbe0790/volumes"
Nov 02 21:58:53.155785 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:53.155748    4929 generic.go:356] "Generic (PLEG): container finished" podID="f7c60948-0876-4c3c-822e-a954bf3f02fb" containerID="e2d2ae9667d7eaaca744425047dca706a06e0cc378e7bc9b653da7306a57be2f" exitCode=0
Nov 02 21:58:53.156162 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:53.155818    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2" event={"ID":"f7c60948-0876-4c3c-822e-a954bf3f02fb","Type":"ContainerDied","Data":"e2d2ae9667d7eaaca744425047dca706a06e0cc378e7bc9b653da7306a57be2f"}
Nov 02 21:58:53.156162 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:53.156139    4929 scope.go:122] "RemoveContainer" containerID="e2d2ae9667d7eaaca744425047dca706a06e0cc378e7bc9b653da7306a57be2f"
Nov 02 21:58:54.160987 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:58:54.160951    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hrts2" event={"ID":"f7c60948-0876-4c3c-822e-a954bf3f02fb","Type":"ContainerStarted","Data":"216504e361110e5d6687e7d22381d9277f4683c9c0470fc4147ef264205529d5"}
Nov 02 21:58:57.494489 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:58:57.494453    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 21:59:14.230417 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:14.230381    4929 generic.go:356] "Generic (PLEG): container finished" podID="418c8848-c1f8-449c-9406-f6c70830cde5" containerID="cc253ed491962efa5978b168408a483c6a0c59b32212af68f3b4bbf2788e225f" exitCode=0
Nov 02 21:59:14.230866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:14.230451    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm" event={"ID":"418c8848-c1f8-449c-9406-f6c70830cde5","Type":"ContainerDied","Data":"cc253ed491962efa5978b168408a483c6a0c59b32212af68f3b4bbf2788e225f"}
Nov 02 21:59:14.230866 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:14.230825    4929 scope.go:122] "RemoveContainer" containerID="cc253ed491962efa5978b168408a483c6a0c59b32212af68f3b4bbf2788e225f"
Nov 02 21:59:15.235212 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:15.235179    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6hbtm" event={"ID":"418c8848-c1f8-449c-9406-f6c70830cde5","Type":"ContainerStarted","Data":"834423bb32b327c1750c62a7ebc8ee185ba7e52e5d3e455a3163e34494c2d55c"}
Nov 02 21:59:17.244794 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:17.244761    4929 generic.go:356] "Generic (PLEG): container finished" podID="c61cf27a-32bb-4a64-9ef4-56badad04b6d" containerID="2fe34936e4418a13b49cd0e09f5d641ec87a868a09f5474f9687b863a7e93615" exitCode=0
Nov 02 21:59:17.245152 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:17.244802    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jx4w4" event={"ID":"c61cf27a-32bb-4a64-9ef4-56badad04b6d","Type":"ContainerDied","Data":"2fe34936e4418a13b49cd0e09f5d641ec87a868a09f5474f9687b863a7e93615"}
Nov 02 21:59:17.245152 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:17.245121    4929 scope.go:122] "RemoveContainer" containerID="2fe34936e4418a13b49cd0e09f5d641ec87a868a09f5474f9687b863a7e93615"
Nov 02 21:59:18.249554 ip-10-0-131-57 kubenswrapper[4929]: I1102 21:59:18.249510    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jx4w4" event={"ID":"c61cf27a-32bb-4a64-9ef4-56badad04b6d","Type":"ContainerStarted","Data":"25da9ca04c5eba243e7bf608fe363e80b0f890e8a6b48d2d9f2d8dcb1b4c2ce6"}
Nov 02 21:59:27.494160 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:59:27.494124    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 21:59:57.494886 ip-10-0-131-57 kubenswrapper[4929]: E1102 21:59:57.494796    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:00:27.495170 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:00:27.495131    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:00:57.495239 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:00:57.495200    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:01:27.493933 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:01:27.493854    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:01:57.494932 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:01:57.494893    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:02:27.495594 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:02:27.495525    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:02:29.308151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308090    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308451    4929 cpu_manager.go:414] "containerMap: removing container" podUID="cd31455a-0dba-4b59-b254-39739cbe0790" containerName="registry"
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308464    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd31455a-0dba-4b59-b254-39739cbe0790" containerName="registry"
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308474    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" containerName="registry"
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308479    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" containerName="registry"
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308550    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd31455a-0dba-4b59-b254-39739cbe0790" containerName="registry"
Nov 02 22:02:29.308639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.308559    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87a2020-99e4-4d23-bd7c-47ed49b6925e" containerName="registry"
Nov 02 22:02:29.311377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.311359    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.314803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.314774    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 22:02:29.314803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.314774    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 22:02:29.314998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.314849    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kzcn8\""
Nov 02 22:02:29.317884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.317626    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 22:02:29.321685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.321660    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 22:02:29.323050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.323028    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:02:29.426212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426174    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426225    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426306    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426351    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4pl\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426421    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426453    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.426485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.426471    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527061 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527027    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527069    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4pl\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527093    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527110    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527128    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527177    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527214    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.527940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.527914    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.529472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.529448    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.529596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.529521    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.529865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.529846    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.529950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.529896    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.534580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.534558    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.534881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.534865    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4pl\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\") pod \"istiod-openshift-gateway-7876bfcb74-zsngn\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.622857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.622822    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:29.894598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.894508    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:02:29.897518 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:02:29.897488    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bd4d36_8e17_4386_a0cd_3a2358528d16.slice/crio-126d2f8615ec5fbaef3d298cf0548398370b0c6992b7126491fdbf47a6efca5d WatchSource:0}: Error finding container 126d2f8615ec5fbaef3d298cf0548398370b0c6992b7126491fdbf47a6efca5d: Status 404 returned error can't find the container with id 126d2f8615ec5fbaef3d298cf0548398370b0c6992b7126491fdbf47a6efca5d
Nov 02 22:02:29.911866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:29.911837    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" event={"ID":"91bd4d36-8e17-4386-a0cd-3a2358528d16","Type":"ContainerStarted","Data":"126d2f8615ec5fbaef3d298cf0548398370b0c6992b7126491fdbf47a6efca5d"}
Nov 02 22:02:33.022039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:33.022004    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:02:33.022375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:33.022078    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:02:33.930036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:33.930003    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" event={"ID":"91bd4d36-8e17-4386-a0cd-3a2358528d16","Type":"ContainerStarted","Data":"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"}
Nov 02 22:02:33.930202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:33.930079    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:33.954034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:33.953987    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" podStartSLOduration=1.832229433 podStartE2EDuration="4.953972481s" totalImagesPullingTime="3.121743048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:33.952258054 +0000 UTC m=+291.260348871" watchObservedRunningTime="2026-11-02 22:02:33.953972481 +0000 UTC m=+291.262063284"
Nov 02 22:02:34.935433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:34.935403    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:02:42.818081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:42.818040    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:02:42.818875 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:02:42.818854    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:02:57.495314 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:02:57.495235    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:03:09.482980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.482942    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"]
Nov 02 22:03:09.491484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.491458    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.495838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.495810    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 22:03:09.495958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.495942    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 22:03:09.496041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.496024    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 22:03:09.496095 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.496048    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 22:03:09.496295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.496277    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8qjb7\""
Nov 02 22:03:09.496372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.496332    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 22:03:09.518662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.518635    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"]
Nov 02 22:03:09.656201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656167    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.656379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656208    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.656379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656301    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.656451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656397    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.656451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656439    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.656513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.656485    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvfg\" (UniqueName: \"kubernetes.io/projected/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-kube-api-access-mxvfg\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757247    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvfg\" (UniqueName: \"kubernetes.io/projected/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-kube-api-access-mxvfg\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757303    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757334    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757368    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757395    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757449    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.757803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.757750    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.759770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.759740    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.760005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.759982    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.760241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.760220    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.760369 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.760350    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.765994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.765969    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvfg\" (UniqueName: \"kubernetes.io/projected/94c958e5-ebd2-47a9-a69d-2eb5db98f6e4-kube-api-access-mxvfg\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-wjktb\" (UID: \"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:09.806807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:09.806771    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:10.105006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:10.104953    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"]
Nov 02 22:03:10.107480 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:03:10.107450    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c958e5_ebd2_47a9_a69d_2eb5db98f6e4.slice/crio-7573b221a3ccf89e8664dbd870661cf8856c62302298288e1ff243100186716d WatchSource:0}: Error finding container 7573b221a3ccf89e8664dbd870661cf8856c62302298288e1ff243100186716d: Status 404 returned error can't find the container with id 7573b221a3ccf89e8664dbd870661cf8856c62302298288e1ff243100186716d
Nov 02 22:03:10.109322 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:10.109300    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:03:11.061279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:11.061241    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb" event={"ID":"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4","Type":"ContainerStarted","Data":"7573b221a3ccf89e8664dbd870661cf8856c62302298288e1ff243100186716d"}
Nov 02 22:03:16.082425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:16.082384    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb" event={"ID":"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4","Type":"ContainerStarted","Data":"26c4776e3ef5caced93aa7c8def3b5d4ca8193e65e665dc9b173d0f2c40903ac"}
Nov 02 22:03:17.087321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:17.087281    4929 generic.go:356] "Generic (PLEG): container finished" podID="94c958e5-ebd2-47a9-a69d-2eb5db98f6e4" containerID="26c4776e3ef5caced93aa7c8def3b5d4ca8193e65e665dc9b173d0f2c40903ac" exitCode=0
Nov 02 22:03:17.087806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:17.087330    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb" event={"ID":"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4","Type":"ContainerDied","Data":"26c4776e3ef5caced93aa7c8def3b5d4ca8193e65e665dc9b173d0f2c40903ac"}
Nov 02 22:03:18.094162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:18.094118    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb" event={"ID":"94c958e5-ebd2-47a9-a69d-2eb5db98f6e4","Type":"ContainerStarted","Data":"3a74b0dfabdec0b6a6d05c1583fe93122ede475d231e2b9fc3e4d7312fc209d1"}
Nov 02 22:03:18.094595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:18.094270    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:18.132173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:18.132082    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb" podStartSLOduration=2.455567784 podStartE2EDuration="9.132067977s" totalImagesPullingTime="6.257653299s" totalInitContainerRuntime="418.846894ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:18.130289487 +0000 UTC m=+335.438380293" watchObservedRunningTime="2026-11-02 22:03:18.132067977 +0000 UTC m=+335.440158833"
Nov 02 22:03:27.494764 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:03:27.494724    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:03:29.101294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:29.101263    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-wjktb"
Nov 02 22:03:50.705674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.705627    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"]
Nov 02 22:03:50.713324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.713298    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.732360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.732326    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"]
Nov 02 22:03:50.766923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.766880    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 22:03:50.767141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.766947    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 22:03:50.767141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.766960    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m49qh\""
Nov 02 22:03:50.800867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.800834    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pkb\" (UniqueName: \"kubernetes.io/projected/2cf18102-5f02-4e2c-b46b-043328bb8f66-kube-api-access-p6pkb\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.801070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.800883    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf18102-5f02-4e2c-b46b-043328bb8f66-tmp\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.801070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.800975    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf18102-5f02-4e2c-b46b-043328bb8f66-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.901475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.901435    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pkb\" (UniqueName: \"kubernetes.io/projected/2cf18102-5f02-4e2c-b46b-043328bb8f66-kube-api-access-p6pkb\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.901675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.901492    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf18102-5f02-4e2c-b46b-043328bb8f66-tmp\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.901675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.901529    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf18102-5f02-4e2c-b46b-043328bb8f66-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.903790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.903763    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf18102-5f02-4e2c-b46b-043328bb8f66-tmp\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.904004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.903984    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf18102-5f02-4e2c-b46b-043328bb8f66-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:50.910760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:50.910733    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pkb\" (UniqueName: \"kubernetes.io/projected/2cf18102-5f02-4e2c-b46b-043328bb8f66-kube-api-access-p6pkb\") pod \"kube-auth-proxy-5747dfd5df-mtdsj\" (UID: \"2cf18102-5f02-4e2c-b46b-043328bb8f66\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:51.026443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:51.026358    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"
Nov 02 22:03:51.311874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:51.311800    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj"]
Nov 02 22:03:51.319218 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:03:51.319187    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf18102_5f02_4e2c_b46b_043328bb8f66.slice/crio-b3d859facccc61e15c611867e0e7aad42374ae4c81150a0ec25172d2f948d100 WatchSource:0}: Error finding container b3d859facccc61e15c611867e0e7aad42374ae4c81150a0ec25172d2f948d100: Status 404 returned error can't find the container with id b3d859facccc61e15c611867e0e7aad42374ae4c81150a0ec25172d2f948d100
Nov 02 22:03:52.237805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:52.237753    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj" event={"ID":"2cf18102-5f02-4e2c-b46b-043328bb8f66","Type":"ContainerStarted","Data":"b3d859facccc61e15c611867e0e7aad42374ae4c81150a0ec25172d2f948d100"}
Nov 02 22:03:54.248567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:54.248458    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj" event={"ID":"2cf18102-5f02-4e2c-b46b-043328bb8f66","Type":"ContainerStarted","Data":"a8b75eff9ee7ffd7b6298e7969d9e755af300a636b9076d35659994975884236"}
Nov 02 22:03:54.265948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:54.265886    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-mtdsj" podStartSLOduration=1.678637055 podStartE2EDuration="4.265869045s" totalImagesPullingTime="2.58723199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:54.26460714 +0000 UTC m=+371.572697945" watchObservedRunningTime="2026-11-02 22:03:54.265869045 +0000 UTC m=+371.573959850"
Nov 02 22:03:57.496008 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:03:57.495967    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:03:57.810265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.810180    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:03:57.813758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.813738    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:57.820915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.820559    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:03:57.823221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.823197    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjchw\""
Nov 02 22:03:57.844181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.844142    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 22:03:57.844181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.844156    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 22:03:57.844385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.844188    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 22:03:57.847707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.847679    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:03:57.961619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.961583    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbx7\" (UniqueName: \"kubernetes.io/projected/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-kube-api-access-drbx7\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:57.961619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:57.961621    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-config-file\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.062972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.062882    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbx7\" (UniqueName: \"kubernetes.io/projected/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-kube-api-access-drbx7\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.062972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.062923    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-config-file\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.063623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.063604    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-config-file\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.070778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.070753    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbx7\" (UniqueName: \"kubernetes.io/projected/e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3-kube-api-access-drbx7\") pod \"limitador-limitador-7c9b87d574-jpmtr\" (UID: \"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.127227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.127185    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:03:58.397576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:58.397523    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:03:58.400426 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:03:58.400394    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ac2a3a_5b67_4a4f_a0b3_eafcd7d8dbf3.slice/crio-8100d5783c41849fc67badb950c2272c031712b1eae2aad59924c699cef92ab0 WatchSource:0}: Error finding container 8100d5783c41849fc67badb950c2272c031712b1eae2aad59924c699cef92ab0: Status 404 returned error can't find the container with id 8100d5783c41849fc67badb950c2272c031712b1eae2aad59924c699cef92ab0
Nov 02 22:03:59.270761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:03:59.270722    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr" event={"ID":"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3","Type":"ContainerStarted","Data":"8100d5783c41849fc67badb950c2272c031712b1eae2aad59924c699cef92ab0"}
Nov 02 22:04:01.281655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:01.281556    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr" event={"ID":"e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3","Type":"ContainerStarted","Data":"8b702ec8d71243cb98a0e404e0d7def5b6d8eaecd370059fdc834be7d08e1b80"}
Nov 02 22:04:01.282048 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:01.281668    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:04:12.286610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:12.286576    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"
Nov 02 22:04:12.309124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:12.309060    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpmtr" podStartSLOduration=12.680485029 podStartE2EDuration="15.309046648s" totalImagesPullingTime="2.628561619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:01.29708984 +0000 UTC m=+378.605180644" watchObservedRunningTime="2026-11-02 22:04:12.309046648 +0000 UTC m=+389.617137504"
Nov 02 22:04:27.494649 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:04:27.494567    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:04:47.045285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.045175    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:04:47.055039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.055011    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.069164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.069132    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:04:47.155086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155051    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155104    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155191    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155234    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsnp5\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155271    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155491 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155296    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.155491 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.155376    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255747    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255801    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255835    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255855    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255872    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsnp5\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255891    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.255995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.255906    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.256697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.256671    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.258354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.258323    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.258456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.258363    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.259120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.259093    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.259120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.259108    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.263237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.263219    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsnp5\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.265891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.265873    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8bzv\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.367979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.367937    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:47.653654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.653597    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:04:47.656305 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:04:47.656273    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0187bc2b_a254_4f14_a4cc_36b8c8be53c5.slice/crio-d756aeda60bdf81efefd04a77ba6bc0022c1faeac242c4fee992904d62e1d454 WatchSource:0}: Error finding container d756aeda60bdf81efefd04a77ba6bc0022c1faeac242c4fee992904d62e1d454: Status 404 returned error can't find the container with id d756aeda60bdf81efefd04a77ba6bc0022c1faeac242c4fee992904d62e1d454
Nov 02 22:04:47.658513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.658477    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:04:47.658616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:47.658565    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:04:48.484319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.484281    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerStarted","Data":"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"}
Nov 02 22:04:48.484864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.484331    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerStarted","Data":"d756aeda60bdf81efefd04a77ba6bc0022c1faeac242c4fee992904d62e1d454"}
Nov 02 22:04:48.484864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.484488    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:48.486278 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.486251    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:04:48.508951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.508872    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" podStartSLOduration=1.508857991 podStartE2EDuration="1.508857991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:48.506522564 +0000 UTC m=+425.814613370" watchObservedRunningTime="2026-11-02 22:04:48.508857991 +0000 UTC m=+425.816948796"
Nov 02 22:04:48.552521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.552484    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:04:48.552782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.552738    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" podUID="91bd4d36-8e17-4386-a0cd-3a2358528d16" containerName="discovery" containerID="cri-o://de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9" gracePeriod=30
Nov 02 22:04:48.804742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.804712    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:04:48.869774 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869738    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.869939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869778    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.869939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869810    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.869939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869863    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.869939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869922    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.870149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869952    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.870149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.869988    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\") pod \"91bd4d36-8e17-4386-a0cd-3a2358528d16\" (UID: \"91bd4d36-8e17-4386-a0cd-3a2358528d16\") "
Nov 02 22:04:48.870261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.870210    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:04:48.870362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.870322    4929 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-ca-configmap\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.872361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872331    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:04:48.872566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872524    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:04:48.872640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872546    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:04:48.872846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872812    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "kube-api-access-pf4pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:04:48.872932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872866    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:04:48.872932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.872894    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token" pod "91bd4d36-8e17-4386-a0cd-3a2358528d16" (UID: "91bd4d36-8e17-4386-a0cd-3a2358528d16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:04:48.970897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970860    4929 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bd4d36-8e17-4386-a0cd-3a2358528d16-local-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.970897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970892    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4pl\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-kube-api-access-pf4pl\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.970897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970904    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.971133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970913    4929 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-kubeconfig\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.971133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970922    4929 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-cacerts\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:48.971133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:48.970930    4929 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91bd4d36-8e17-4386-a0cd-3a2358528d16-istio-csr-dns-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:49.491442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.491405    4929 generic.go:356] "Generic (PLEG): container finished" podID="91bd4d36-8e17-4386-a0cd-3a2358528d16" containerID="de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9" exitCode=0
Nov 02 22:04:49.491890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.491455    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"
Nov 02 22:04:49.491890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.491489    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" event={"ID":"91bd4d36-8e17-4386-a0cd-3a2358528d16","Type":"ContainerDied","Data":"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"}
Nov 02 22:04:49.491890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.491559    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn" event={"ID":"91bd4d36-8e17-4386-a0cd-3a2358528d16","Type":"ContainerDied","Data":"126d2f8615ec5fbaef3d298cf0548398370b0c6992b7126491fdbf47a6efca5d"}
Nov 02 22:04:49.491890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.491587    4929 scope.go:122] "RemoveContainer" containerID="de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"
Nov 02 22:04:49.500847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.500829    4929 scope.go:122] "RemoveContainer" containerID="de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"
Nov 02 22:04:49.501115 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:04:49.501096    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9\": container with ID starting with de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9 not found: ID does not exist" containerID="de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"
Nov 02 22:04:49.501161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.501124    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9"} err="failed to get container status \"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9\": rpc error: code = NotFound desc = could not find container \"de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9\": container with ID starting with de4ec17eac292d4d0408f7ac6d4eb3eb1eaf44ea34d1d42b4d71780748ffe5e9 not found: ID does not exist"
Nov 02 22:04:49.513054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.513031    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:04:49.516693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:49.516670    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsngn"]
Nov 02 22:04:50.802027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:04:50.801985    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bd4d36-8e17-4386-a0cd-3a2358528d16" path="/var/lib/kubelet/pods/91bd4d36-8e17-4386-a0cd-3a2358528d16/volumes"
Nov 02 22:04:57.495294 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:04:57.495252    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:05:13.620507 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.620470    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:13.621012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.620793    4929 cpu_manager.go:414] "containerMap: removing container" podUID="91bd4d36-8e17-4386-a0cd-3a2358528d16" containerName="discovery"
Nov 02 22:05:13.621012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.620805    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bd4d36-8e17-4386-a0cd-3a2358528d16" containerName="discovery"
Nov 02 22:05:13.621012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.620879    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bd4d36-8e17-4386-a0cd-3a2358528d16" containerName="discovery"
Nov 02 22:05:13.624090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.624073    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.633708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.633679    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:05:13.633864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.633806    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 22:05:13.633864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.633839    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 22:05:13.633989 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.633928    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tcmbk\""
Nov 02 22:05:13.635056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.635033    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:13.635144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.635130    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:05:13.784451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.784406    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.784647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.784465    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.784647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.784585    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.784647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.784619    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknqv\" (UniqueName: \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.885972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.885875    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.885972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.885939    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.885972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.885969    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.886254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.885995    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknqv\" (UniqueName: \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.886925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.886875    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.889011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.888982    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.889146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.888984    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.894386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.894364    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknqv\" (UniqueName: \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\") pod \"maas-api-6d669c7fcd-mb5gv\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:13.937205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:13.937167    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:14.208594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:14.208569    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:14.211464 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:05:14.211432    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77beb9bc_101d_4948_91f0_6f620e02614e.slice/crio-d261431dd96c6fd38bb45b5e29e38d5232229c624389b818c891a30995251ae5 WatchSource:0}: Error finding container d261431dd96c6fd38bb45b5e29e38d5232229c624389b818c891a30995251ae5: Status 404 returned error can't find the container with id d261431dd96c6fd38bb45b5e29e38d5232229c624389b818c891a30995251ae5
Nov 02 22:05:14.403213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:14.403179    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:05:14.591938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:14.591854    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" event={"ID":"77beb9bc-101d-4948-91f0-6f620e02614e","Type":"ContainerStarted","Data":"d261431dd96c6fd38bb45b5e29e38d5232229c624389b818c891a30995251ae5"}
Nov 02 22:05:17.607551 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:17.607505    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" event={"ID":"77beb9bc-101d-4948-91f0-6f620e02614e","Type":"ContainerStarted","Data":"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"}
Nov 02 22:05:17.607950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:17.607730    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:17.624516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:17.624458    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" podStartSLOduration=1.963700502 podStartE2EDuration="4.624439038s" totalImagesPullingTime="2.660738536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:17.623664949 +0000 UTC m=+454.931755846" watchObservedRunningTime="2026-11-02 22:05:17.624439038 +0000 UTC m=+454.932529844"
Nov 02 22:05:23.617004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:23.616965    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:24.495262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.495222    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:05:24.501617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.501590    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.506073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.506047    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:05:24.562114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.562070    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.562282 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.562126    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.562282 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.562248    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.562360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.562281    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6r4\" (UniqueName: \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.663051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.663017    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.663497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.663066    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.663497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.663101    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.663497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.663128    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6r4\" (UniqueName: \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.663936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.663911    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.665577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.665555    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.665665 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.665644    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.676424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.676401    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6r4\" (UniqueName: \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\") pod \"maas-api-746984ddc4-x9q9t\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:24.817030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:24.816949    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:25.089521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:25.089438    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:05:25.092612 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:05:25.092584    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac17491a_49fb_42c8_873d_c4d4a457a2fd.slice/crio-70c5ab3ad5c314652886b48cceb03cb6da910e3ca826720d5e63df33047d873d WatchSource:0}: Error finding container 70c5ab3ad5c314652886b48cceb03cb6da910e3ca826720d5e63df33047d873d: Status 404 returned error can't find the container with id 70c5ab3ad5c314652886b48cceb03cb6da910e3ca826720d5e63df33047d873d
Nov 02 22:05:25.643926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:25.643890    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" event={"ID":"ac17491a-49fb-42c8-873d-c4d4a457a2fd","Type":"ContainerStarted","Data":"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"}
Nov 02 22:05:25.644118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:25.643940    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" event={"ID":"ac17491a-49fb-42c8-873d-c4d4a457a2fd","Type":"ContainerStarted","Data":"70c5ab3ad5c314652886b48cceb03cb6da910e3ca826720d5e63df33047d873d"}
Nov 02 22:05:25.644118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:25.643996    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:25.660447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:25.660396    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" podStartSLOduration=1.222611413 podStartE2EDuration="1.660382791s" totalImagesPullingTime="437.771378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:25.659171932 +0000 UTC m=+462.967262737" watchObservedRunningTime="2026-11-02 22:05:25.660382791 +0000 UTC m=+462.968473618"
Nov 02 22:05:27.495902 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:05:27.495860    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:05:31.653236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:31.653207    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:05:31.704387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:31.704347    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:31.704623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:31.704606    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" podUID="77beb9bc-101d-4948-91f0-6f620e02614e" containerName="maas-api" containerID="cri-o://bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f" gracePeriod=30
Nov 02 22:05:31.959158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:31.959134    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:32.019228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.019189    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\") pod \"77beb9bc-101d-4948-91f0-6f620e02614e\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") "
Nov 02 22:05:32.019423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.019249    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\") pod \"77beb9bc-101d-4948-91f0-6f620e02614e\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") "
Nov 02 22:05:32.019423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.019276    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\" (UniqueName: \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\") pod \"77beb9bc-101d-4948-91f0-6f620e02614e\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") "
Nov 02 22:05:32.019423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.019407    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\") pod \"77beb9bc-101d-4948-91f0-6f620e02614e\" (UID: \"77beb9bc-101d-4948-91f0-6f620e02614e\") "
Nov 02 22:05:32.019691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.019664    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca" pod "77beb9bc-101d-4948-91f0-6f620e02614e" (UID: "77beb9bc-101d-4948-91f0-6f620e02614e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:05:32.021674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.021645    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv" pod "77beb9bc-101d-4948-91f0-6f620e02614e" (UID: "77beb9bc-101d-4948-91f0-6f620e02614e"). InnerVolumeSpecName "kube-api-access-zknqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:05:32.021766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.021645    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls" pod "77beb9bc-101d-4948-91f0-6f620e02614e" (UID: "77beb9bc-101d-4948-91f0-6f620e02614e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:05:32.021766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.021719    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs" pod "77beb9bc-101d-4948-91f0-6f620e02614e" (UID: "77beb9bc-101d-4948-91f0-6f620e02614e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:05:32.120231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.120190    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:32.120231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.120224    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77beb9bc-101d-4948-91f0-6f620e02614e-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:32.120231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.120233    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77beb9bc-101d-4948-91f0-6f620e02614e-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:32.120467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.120244    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknqv\" (UniqueName: \"kubernetes.io/projected/77beb9bc-101d-4948-91f0-6f620e02614e-kube-api-access-zknqv\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:32.673090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.673055    4929 generic.go:356] "Generic (PLEG): container finished" podID="77beb9bc-101d-4948-91f0-6f620e02614e" containerID="bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f" exitCode=0
Nov 02 22:05:32.673493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.673098    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"
Nov 02 22:05:32.673493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.673140    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" event={"ID":"77beb9bc-101d-4948-91f0-6f620e02614e","Type":"ContainerDied","Data":"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"}
Nov 02 22:05:32.673493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.673190    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv" event={"ID":"77beb9bc-101d-4948-91f0-6f620e02614e","Type":"ContainerDied","Data":"d261431dd96c6fd38bb45b5e29e38d5232229c624389b818c891a30995251ae5"}
Nov 02 22:05:32.673493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.673211    4929 scope.go:122] "RemoveContainer" containerID="bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"
Nov 02 22:05:32.682825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.682801    4929 scope.go:122] "RemoveContainer" containerID="bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"
Nov 02 22:05:32.683125 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:05:32.683104    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f\": container with ID starting with bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f not found: ID does not exist" containerID="bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"
Nov 02 22:05:32.683207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.683131    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f"} err="failed to get container status \"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f\": rpc error: code = NotFound desc = could not find container \"bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f\": container with ID starting with bc394710421788288aee03a8d5ffa732a94b4e17d04fcba4c0d04d1f12d0f35f not found: ID does not exist"
Nov 02 22:05:32.696258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.696229    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:32.699728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.699705    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mb5gv"]
Nov 02 22:05:32.802549 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:05:32.802508    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77beb9bc-101d-4948-91f0-6f620e02614e" path="/var/lib/kubelet/pods/77beb9bc-101d-4948-91f0-6f620e02614e/volumes"
Nov 02 22:05:57.496054 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:05:57.495979    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:06:27.495094 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:06:27.495055    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:06:41.940938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.940905    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"]
Nov 02 22:06:41.941338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.941216    4929 cpu_manager.go:414] "containerMap: removing container" podUID="77beb9bc-101d-4948-91f0-6f620e02614e" containerName="maas-api"
Nov 02 22:06:41.941338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.941226    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77beb9bc-101d-4948-91f0-6f620e02614e" containerName="maas-api"
Nov 02 22:06:41.941338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.941311    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="77beb9bc-101d-4948-91f0-6f620e02614e" containerName="maas-api"
Nov 02 22:06:41.944437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.944420    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:41.952810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.952443    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"]
Nov 02 22:06:41.952810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.952713    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bf7n2\""
Nov 02 22:06:41.952810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.952793    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 22:06:41.952990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.952910    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 22:06:41.952990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:41.952949    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 22:06:42.028190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028154    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.028360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028199    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszvk\" (UniqueName: \"kubernetes.io/projected/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-kube-api-access-dszvk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.028360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028285    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.028360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028321    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.028360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028343    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.028495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.028427    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129641    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129691    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszvk\" (UniqueName: \"kubernetes.io/projected/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-kube-api-access-dszvk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129730    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129748    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129785    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.129928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.129828    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.130182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.130155    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.130316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.130215    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.130316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.130236    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.131930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.131908    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.132239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.132221    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.137392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.137374    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszvk\" (UniqueName: \"kubernetes.io/projected/bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70-kube-api-access-dszvk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj\" (UID: \"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.258814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.258725    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:42.327388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.327356    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:06:42.528785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.528708    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"]
Nov 02 22:06:42.532021 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:06:42.531985    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8cdbfe_4ab5_4aa8_84de_f25c7a6bbb70.slice/crio-37f7ee870e7249671c8c997d58e041a67b379c57d96f9bf2d198ee019fc5d6cd WatchSource:0}: Error finding container 37f7ee870e7249671c8c997d58e041a67b379c57d96f9bf2d198ee019fc5d6cd: Status 404 returned error can't find the container with id 37f7ee870e7249671c8c997d58e041a67b379c57d96f9bf2d198ee019fc5d6cd
Nov 02 22:06:42.953143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:42.953102    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" event={"ID":"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70","Type":"ContainerStarted","Data":"37f7ee870e7249671c8c997d58e041a67b379c57d96f9bf2d198ee019fc5d6cd"}
Nov 02 22:06:44.813589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:44.813558    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:06:44.966117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:44.966074    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" event={"ID":"bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70","Type":"ContainerStarted","Data":"2c64e236671bf5472bbc016df1daca16ea267f43f1e2e4a715a36475084ce31b"}
Nov 02 22:06:45.003405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:45.003345    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" podStartSLOduration=2.587667136 podStartE2EDuration="4.003331278s" totalImagesPullingTime="1.415664142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:45.001949301 +0000 UTC m=+542.310040109" watchObservedRunningTime="2026-11-02 22:06:45.003331278 +0000 UTC m=+542.311422080"
Nov 02 22:06:45.259456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:45.259415    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:46.689226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.689192    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"]
Nov 02 22:06:46.692789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.692764    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.697151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.697129    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 22:06:46.701927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.701903    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"]
Nov 02 22:06:46.768908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.768876    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.768908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.768910    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.769138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.768933    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.769138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.769089    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.769138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.769126    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.769250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.769146    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlx8j\" (UniqueName: \"kubernetes.io/projected/7a4a8e4d-2131-4854-9d0b-5010dae227bb-kube-api-access-xlx8j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.869983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.869947    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.869983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.869983    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.869999    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870061    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870084    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870120    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlx8j\" (UniqueName: \"kubernetes.io/projected/7a4a8e4d-2131-4854-9d0b-5010dae227bb-kube-api-access-xlx8j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870421    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870460    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.870736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.870702    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.872405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.872380    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a8e4d-2131-4854-9d0b-5010dae227bb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.872638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.872621    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a8e4d-2131-4854-9d0b-5010dae227bb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:46.878233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:46.878209    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlx8j\" (UniqueName: \"kubernetes.io/projected/7a4a8e4d-2131-4854-9d0b-5010dae227bb-kube-api-access-xlx8j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-svvlm\" (UID: \"7a4a8e4d-2131-4854-9d0b-5010dae227bb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:47.007575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.007480    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:47.198406 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.198367    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:06:47.281345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.281275    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"]
Nov 02 22:06:47.284240 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:06:47.284206    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4a8e4d_2131_4854_9d0b_5010dae227bb.slice/crio-8f95ad613346a5739c446410a680adc2de8fa9edf2602045430748279c8c95dd WatchSource:0}: Error finding container 8f95ad613346a5739c446410a680adc2de8fa9edf2602045430748279c8c95dd: Status 404 returned error can't find the container with id 8f95ad613346a5739c446410a680adc2de8fa9edf2602045430748279c8c95dd
Nov 02 22:06:47.777810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.777777    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"]
Nov 02 22:06:47.781357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.781332    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.788868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.788835    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 22:06:47.793201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.793174    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"]
Nov 02 22:06:47.879793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.879758    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.879946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.879799    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5hj\" (UniqueName: \"kubernetes.io/projected/8b76db34-a58f-4b26-91c1-d5c678aec811-kube-api-access-lf5hj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.879946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.879826    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b76db34-a58f-4b26-91c1-d5c678aec811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.880017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.879951    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.880017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.879996    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.880086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.880071    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980497    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980574    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5hj\" (UniqueName: \"kubernetes.io/projected/8b76db34-a58f-4b26-91c1-d5c678aec811-kube-api-access-lf5hj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980632    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b76db34-a58f-4b26-91c1-d5c678aec811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980667    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980689    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.980725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.980721    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.981139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.981105    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.981139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.981122    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.981348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.981192    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.981676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.981647    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" event={"ID":"7a4a8e4d-2131-4854-9d0b-5010dae227bb","Type":"ContainerStarted","Data":"5498a2938bac19fa270e87b50d068f3686b19076c9d5187214b673cf90f2a81d"}
Nov 02 22:06:47.981768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.981708    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" event={"ID":"7a4a8e4d-2131-4854-9d0b-5010dae227bb","Type":"ContainerStarted","Data":"8f95ad613346a5739c446410a680adc2de8fa9edf2602045430748279c8c95dd"}
Nov 02 22:06:47.982971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.982946    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b76db34-a58f-4b26-91c1-d5c678aec811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.983401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.983386    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b76db34-a58f-4b26-91c1-d5c678aec811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.988348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.988327    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5hj\" (UniqueName: \"kubernetes.io/projected/8b76db34-a58f-4b26-91c1-d5c678aec811-kube-api-access-lf5hj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc\" (UID: \"8b76db34-a58f-4b26-91c1-d5c678aec811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:47.997374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:47.997336    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" podStartSLOduration=1.6841946490000002 podStartE2EDuration="1.997322316s" totalImagesPullingTime="313.127667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:47.996652011 +0000 UTC m=+545.304742827" watchObservedRunningTime="2026-11-02 22:06:47.997322316 +0000 UTC m=+545.305413121"
Nov 02 22:06:48.007849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:48.007829    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:48.095172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:48.095135    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:48.365849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:48.365805    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"]
Nov 02 22:06:48.368384 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:06:48.368354    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76db34_a58f_4b26_91c1_d5c678aec811.slice/crio-94c5cc2ccae8ac8972f81e0e7a3ab5c1c10f64c0f5a0549c5e130b6fdce8c194 WatchSource:0}: Error finding container 94c5cc2ccae8ac8972f81e0e7a3ab5c1c10f64c0f5a0549c5e130b6fdce8c194: Status 404 returned error can't find the container with id 94c5cc2ccae8ac8972f81e0e7a3ab5c1c10f64c0f5a0549c5e130b6fdce8c194
Nov 02 22:06:48.987579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:48.987470    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc" event={"ID":"8b76db34-a58f-4b26-91c1-d5c678aec811","Type":"ContainerStarted","Data":"e79e02c434a399f732460072ec273d1aa954fb0bf8166d1e492eb23c117ee900"}
Nov 02 22:06:48.987579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:48.987556    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc" event={"ID":"8b76db34-a58f-4b26-91c1-d5c678aec811","Type":"ContainerStarted","Data":"94c5cc2ccae8ac8972f81e0e7a3ab5c1c10f64c0f5a0549c5e130b6fdce8c194"}
Nov 02 22:06:49.007718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:49.007597    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc" podStartSLOduration=1.679825709 podStartE2EDuration="2.00758422s" totalImagesPullingTime="327.758511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:49.007091462 +0000 UTC m=+546.315182282" watchObservedRunningTime="2026-11-02 22:06:49.00758422 +0000 UTC m=+546.315675023"
Nov 02 22:06:49.096006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:49.095977    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:50.544520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:50.544486    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:06:51.820489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.820453    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"]
Nov 02 22:06:51.824010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.823988    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.832568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.832509    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 22:06:51.834213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.834181    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"]
Nov 02 22:06:51.916555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916500    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c137d510-9d3e-4e3b-94ec-8daf581918f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.916727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916566    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.916727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916699    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkgv\" (UniqueName: \"kubernetes.io/projected/c137d510-9d3e-4e3b-94ec-8daf581918f9-kube-api-access-jqkgv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.916807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916756    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.916901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916882    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:51.916980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:51.916917    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017603    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017648    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017673    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017705    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c137d510-9d3e-4e3b-94ec-8daf581918f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017727    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.017861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.017776    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkgv\" (UniqueName: \"kubernetes.io/projected/c137d510-9d3e-4e3b-94ec-8daf581918f9-kube-api-access-jqkgv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.018113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.018090    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.018158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.018117    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.018199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.018169    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.020040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.020020    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c137d510-9d3e-4e3b-94ec-8daf581918f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.020334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.020310    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c137d510-9d3e-4e3b-94ec-8daf581918f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.025384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.025358    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkgv\" (UniqueName: \"kubernetes.io/projected/c137d510-9d3e-4e3b-94ec-8daf581918f9-kube-api-access-jqkgv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w\" (UID: \"c137d510-9d3e-4e3b-94ec-8daf581918f9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.149336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.149296    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:52.259474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.259437    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:52.271688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.271658    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:52.420606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:52.420581    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"]
Nov 02 22:06:52.423146 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:06:52.423109    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc137d510_9d3e_4e3b_94ec_8daf581918f9.slice/crio-8a385e5cee1249d27cbd888483dd169b72c8c1b71d513fc4198e2a61aed69d75 WatchSource:0}: Error finding container 8a385e5cee1249d27cbd888483dd169b72c8c1b71d513fc4198e2a61aed69d75: Status 404 returned error can't find the container with id 8a385e5cee1249d27cbd888483dd169b72c8c1b71d513fc4198e2a61aed69d75
Nov 02 22:06:53.007425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:53.007327    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" event={"ID":"c137d510-9d3e-4e3b-94ec-8daf581918f9","Type":"ContainerStarted","Data":"e10a35f69d1a01e5d376d386900aa9f06e980b8923ecdb94d8d8feb4ce238910"}
Nov 02 22:06:53.007425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:53.007385    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" event={"ID":"c137d510-9d3e-4e3b-94ec-8daf581918f9","Type":"ContainerStarted","Data":"8a385e5cee1249d27cbd888483dd169b72c8c1b71d513fc4198e2a61aed69d75"}
Nov 02 22:06:53.019809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:53.019783    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj"
Nov 02 22:06:53.024965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:53.024910    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podStartSLOduration=1.704394513 podStartE2EDuration="2.024897254s" totalImagesPullingTime="320.502741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:53.023581474 +0000 UTC m=+550.331672284" watchObservedRunningTime="2026-11-02 22:06:53.024897254 +0000 UTC m=+550.332988058"
Nov 02 22:06:53.150340 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:53.150306    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:06:55.875619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:55.875583    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:06:57.007909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:57.007867    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:57.020689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:57.020664    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:57.035311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:57.035282    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:06:57.495436 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:06:57.495402    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:06:58.095407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:58.095366    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:58.107594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:58.107566    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:06:59.044663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:06:59.044624    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc"
Nov 02 22:07:02.150287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:02.150251    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:07:02.163196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:02.163172    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:07:03.061717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:03.061688    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:07:19.118907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.118873    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"]
Nov 02 22:07:19.125846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.125823    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.131367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.131330    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"]
Nov 02 22:07:19.133241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.133217    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mctrp\""
Nov 02 22:07:19.138367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.138340    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"]
Nov 02 22:07:19.142649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.142629    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.154277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.154254    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"]
Nov 02 22:07:19.158421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158391    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158632    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158674    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158700    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158796    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158841    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.158925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.158913    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjhm\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-kube-api-access-mzjhm\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.159058 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.159030    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a73ff87-2b7d-4c69-82e2-495f4b866345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.159108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.159092    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.259845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.259808    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.259845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.259850    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.259879    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a73ff87-2b7d-4c69-82e2-495f4b866345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.259959    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c4566-edc7-453d-bd8c-d14867259b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260022    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260063    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260104    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260144    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260173    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260275    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260299    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260318    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260352    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260401    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260440    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260473    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjhm\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-kube-api-access-mzjhm\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260498    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxn8\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-kube-api-access-bgxn8\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260501    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260559    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c4566-edc7-453d-bd8c-d14867259b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.260685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260575    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260717    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260756    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a73ff87-2b7d-4c69-82e2-495f4b866345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.260972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.260816    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.262627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.262605    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.263123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.263098    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.268091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.268069    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjhm\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-kube-api-access-mzjhm\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.268167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.268121    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a73ff87-2b7d-4c69-82e2-495f4b866345-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gjn6k\" (UID: \"4a73ff87-2b7d-4c69-82e2-495f4b866345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.361122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361084    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361129    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxn8\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-kube-api-access-bgxn8\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361150    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c4566-edc7-453d-bd8c-d14867259b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361168    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361200    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361237    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c4566-edc7-453d-bd8c-d14867259b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361276    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361309    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361349    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361666    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361728    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361778    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.361898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.361839    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.362187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.362168    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c4566-edc7-453d-bd8c-d14867259b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.363817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.363793    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c4566-edc7-453d-bd8c-d14867259b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.363993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.363976    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c4566-edc7-453d-bd8c-d14867259b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.368235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.368205    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.368404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.368386    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxn8\" (UniqueName: \"kubernetes.io/projected/047c4566-edc7-453d-bd8c-d14867259b93-kube-api-access-bgxn8\") pod \"maas-default-gateway-openshift-default-696f47c6c-j6cnb\" (UID: \"047c4566-edc7-453d-bd8c-d14867259b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.444988 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.444897    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:19.461302 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.461272    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:19.723347 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.723322    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"]
Nov 02 22:07:19.725728 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:07:19.725701    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a73ff87_2b7d_4c69_82e2_495f4b866345.slice/crio-abcc8345ab37eda358af78e5651c22e532975cb0be8a48bd6fb55dffbecfaf09 WatchSource:0}: Error finding container abcc8345ab37eda358af78e5651c22e532975cb0be8a48bd6fb55dffbecfaf09: Status 404 returned error can't find the container with id abcc8345ab37eda358af78e5651c22e532975cb0be8a48bd6fb55dffbecfaf09
Nov 02 22:07:19.743896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:19.743874    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"]
Nov 02 22:07:19.746284 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:07:19.746258    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047c4566_edc7_453d_bd8c_d14867259b93.slice/crio-41859577aa5c9766a9d197efca87a5a6dcf9150cec14f19e652919cccdddbc68 WatchSource:0}: Error finding container 41859577aa5c9766a9d197efca87a5a6dcf9150cec14f19e652919cccdddbc68: Status 404 returned error can't find the container with id 41859577aa5c9766a9d197efca87a5a6dcf9150cec14f19e652919cccdddbc68
Nov 02 22:07:20.121860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:20.121822    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" event={"ID":"4a73ff87-2b7d-4c69-82e2-495f4b866345","Type":"ContainerStarted","Data":"abcc8345ab37eda358af78e5651c22e532975cb0be8a48bd6fb55dffbecfaf09"}
Nov 02 22:07:20.123001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:20.122979    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" event={"ID":"047c4566-edc7-453d-bd8c-d14867259b93","Type":"ContainerStarted","Data":"41859577aa5c9766a9d197efca87a5a6dcf9150cec14f19e652919cccdddbc68"}
Nov 02 22:07:23.092386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:23.092347    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:23.092784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:23.092435    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:23.092784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:23.092484    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:24.164056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:24.164017    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" event={"ID":"047c4566-edc7-453d-bd8c-d14867259b93","Type":"ContainerStarted","Data":"fff1f4c21d1cc96b1864f6ebe2e8728277a6cbe6e94204fc5032345eea9a00ee"}
Nov 02 22:07:24.184782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:24.184729    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podStartSLOduration=1.841364835 podStartE2EDuration="5.184715077s" totalImagesPullingTime="3.343350242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:24.182714009 +0000 UTC m=+581.490804815" watchObservedRunningTime="2026-11-02 22:07:24.184715077 +0000 UTC m=+581.492805882"
Nov 02 22:07:24.462459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:24.462335    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:24.463815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:24.463776    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:24.463905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:24.463838    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:25.462225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:25.462183    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:25.462747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:25.462243    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:26.420452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:26.420415    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:26.420523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:26.420495    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:26.420583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:26.420527    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:26.462688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:26.462655    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:26.463707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:26.463673    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:27.189181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.189143    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" event={"ID":"4a73ff87-2b7d-4c69-82e2-495f4b866345","Type":"ContainerStarted","Data":"9beb635a4f6fbc9fd5c3a70919770bb4795a169ac1cbe861ae11a035359a1279"}
Nov 02 22:07:27.209436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.209389    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podStartSLOduration=1.518375651 podStartE2EDuration="8.209377468s" totalImagesPullingTime="6.691001817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:27.208727523 +0000 UTC m=+584.516818337" watchObservedRunningTime="2026-11-02 22:07:27.209377468 +0000 UTC m=+584.517468285"
Nov 02 22:07:27.451018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.450929    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:27.465712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.465674    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:27.466300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.466267    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:27.467000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.466978    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:27.467180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:27.467160    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:27.509064 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:07:27.508964    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:07:28.451009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:28.450968    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:28.451297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:28.451263    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:28.463926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:28.463887    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:28.464185 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:28.464160    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:29.450561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:29.448946    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:29.450561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:29.449014    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:29.462649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:29.462609    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:29.462959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:29.462907    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:29.669633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:29.669589    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:07:30.448556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:30.447914    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:30.448556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:30.447984    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:30.461924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:30.461883    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 22:07:30.462430 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:30.461944    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 22:07:31.450591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:31.449960    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:31.450591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:31.450023    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:32.445316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:32.445276    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:32.445754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:32.445344    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:32.463411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:32.463380    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:32.463631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:32.463430    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:33.445817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.445766    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 22:07:33.446344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.445845    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 22:07:33.504696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.504652    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:33.504876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.504708    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:33.868897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.868854    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:33.869307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.869285    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:33.870071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:33.870050    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb"
Nov 02 22:07:34.162716 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.162633    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"]
Nov 02 22:07:34.167441 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.167410    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.177235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.177207    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"]
Nov 02 22:07:34.247088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247052    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247109    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkkf\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-kube-api-access-blkkf\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247279    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247324    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247353    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247374    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247426    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247507    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.247729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.247593    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.348623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348571    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.348817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348650    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.348817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348703    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.348817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348737    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.348817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348790    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkkf\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-kube-api-access-blkkf\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348852    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348925    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.348971    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349002    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349206    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349391 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349340    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349421    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349454    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.349697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.349517    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.351448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.351428    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.351810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.351783    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.356103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.356077    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.356833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.356811    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkkf\" (UniqueName: \"kubernetes.io/projected/5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90-kube-api-access-blkkf\") pod \"maas-default-gateway-openshift-default-696f47c6c-2zhcx\" (UID: \"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.446043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.445943    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:34.490005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.489971    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:34.787078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.787048    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"]
Nov 02 22:07:34.789529 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:07:34.789491    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3f2d6a_0a59_4ad8_a517_4fc8ec0ade90.slice/crio-f40cf9707c3a2e04949e2d57823b54040dbb4c66e8b47a59dc426b22aeeb1c83 WatchSource:0}: Error finding container f40cf9707c3a2e04949e2d57823b54040dbb4c66e8b47a59dc426b22aeeb1c83: Status 404 returned error can't find the container with id f40cf9707c3a2e04949e2d57823b54040dbb4c66e8b47a59dc426b22aeeb1c83
Nov 02 22:07:34.793304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.793266    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:34.793390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.793336    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:34.793390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:34.793378    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:07:35.228983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.228935    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" event={"ID":"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90","Type":"ContainerStarted","Data":"e144ca13293a300b5e7a3711a6a33209729f03cf15c2a4b7240501b638d6d11c"}
Nov 02 22:07:35.229201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.229010    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" event={"ID":"5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90","Type":"ContainerStarted","Data":"f40cf9707c3a2e04949e2d57823b54040dbb4c66e8b47a59dc426b22aeeb1c83"}
Nov 02 22:07:35.249915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.249857    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podStartSLOduration=1.249830595 podStartE2EDuration="1.249830595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:35.247308001 +0000 UTC m=+592.555398808" watchObservedRunningTime="2026-11-02 22:07:35.249830595 +0000 UTC m=+592.557921404"
Nov 02 22:07:35.454375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.453656    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:35.454375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.453767    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:35.491176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.491075    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:35.492599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.492572    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:35.492742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:35.492616    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:36.249559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.236827    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"]
Nov 02 22:07:36.283118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.283084    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"]
Nov 02 22:07:36.283416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.283398    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.383937 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.383894    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 22:07:36.484170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.484136    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbhc\" (UniqueName: \"kubernetes.io/projected/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-kube-api-access-qwbhc\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.485038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.485016    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.490681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.490658    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:36.490892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.490872    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:36.496640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.496616    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:36.496779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.496764    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:36.588060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.587977    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.588292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.588277    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbhc\" (UniqueName: \"kubernetes.io/projected/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-kube-api-access-qwbhc\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.589069 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.589027    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.599317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.599237    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbhc\" (UniqueName: \"kubernetes.io/projected/a5ffeb7d-4629-4724-9c97-a6aca4fbf01b-kube-api-access-qwbhc\") pod \"payload-processing-55fbbbf9dc-r9ggx\" (UID: \"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.602890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.602866    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:36.911190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:36.911132    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"]
Nov 02 22:07:36.914223 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:07:36.914192    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ffeb7d_4629_4724_9c97_a6aca4fbf01b.slice/crio-09d074fc1970857fbfe78cc5416c48e26952533c4d8d8515485a3f3741707f89 WatchSource:0}: Error finding container 09d074fc1970857fbfe78cc5416c48e26952533c4d8d8515485a3f3741707f89: Status 404 returned error can't find the container with id 09d074fc1970857fbfe78cc5416c48e26952533c4d8d8515485a3f3741707f89
Nov 02 22:07:37.278512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:37.278405    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx" event={"ID":"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b","Type":"ContainerStarted","Data":"09d074fc1970857fbfe78cc5416c48e26952533c4d8d8515485a3f3741707f89"}
Nov 02 22:07:37.490899 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:37.490792    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:37.490899 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:37.490862    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:37.491998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:37.491968    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:37.493247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:37.493191    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k"
Nov 02 22:07:38.490898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:38.490868    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:38.491315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:38.490940    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:39.491063 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:39.491027    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:39.491569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:39.491094    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:40.294276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:40.294235    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx" event={"ID":"a5ffeb7d-4629-4724-9c97-a6aca4fbf01b","Type":"ContainerStarted","Data":"524e29c2dd54403d633b8aeabda7e68d7438f0b942a5e2025e1d8c866eff2d88"}
Nov 02 22:07:40.294557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:40.294359    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:40.309956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:40.309907    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx" podStartSLOduration=1.596118158 podStartE2EDuration="4.309888111s" totalImagesPullingTime="2.713769953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:40.309344094 +0000 UTC m=+597.617434897" watchObservedRunningTime="2026-11-02 22:07:40.309888111 +0000 UTC m=+597.617978915"
Nov 02 22:07:40.491283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:40.491243    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:40.491724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:40.491302    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:41.490790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:41.490758    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 22:07:41.490950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:41.490813    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 22:07:42.848342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:42.848309    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:07:42.850169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:42.850144    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:07:43.490374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:43.490334    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:43.490578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:43.490399    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:44.531738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:44.531699    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:07:44.532162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:44.531780    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:07:45.267291 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:45.267252    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:45.267763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:45.267724    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:45.268789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:45.268763    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx"
Nov 02 22:07:51.302585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:07:51.302554    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9ggx"
Nov 02 22:07:57.495521 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:07:57.495485    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:08:27.495938 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:08:27.495897    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:08:28.945995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.945957    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:08:28.949890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.949868    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.964257 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.963506    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2532af-openshift-default-dockercfg-9p29v\""
Nov 02 22:08:28.967179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.967152    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:08:28.985138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985102    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985149    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985245    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985288    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dz9\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985386    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985429    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985464    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985640    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:28.985764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:28.985686    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086291 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086255    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086291 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086303    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086332    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086381    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086411    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086469    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086497    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086523    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086570    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dz9\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.086896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086806    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.087006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086929    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.087006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.086989    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.087124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.087116    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.087276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.087246    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.089068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.089044    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.089461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.089441    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.098904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.098834    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.099218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.099115    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dz9\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.205272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.205191    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:08:29.210108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.210083    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.213789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.213762    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-361a44-openshift-default-dockercfg-6lxp8\""
Nov 02 22:08:29.226597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.226560    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:08:29.263017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.262982    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:29.288255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288213    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288267    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288296    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288484    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288525    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288634    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288683    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288717    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zrn\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.288777 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.288765    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.378129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.378080    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:08:29.382475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.382452    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.390268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390242    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390294    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390387    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390530 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390412    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390530 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390454    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390530 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390487    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390530 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390521    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zrn\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390588    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390624    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.390971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390939    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.391035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.390994    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.392744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.391235    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.392744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.391472    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.392744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.391691    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.394190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.394143    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.394855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.394782    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:08:29.395907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.395865    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.401066 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.401043    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.401685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.401664    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zrn\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f44ds\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.408276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.407522    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-310bb3-openshift-default-dockercfg-qrwjr\""
Nov 02 22:08:29.492146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492071    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492111    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492137    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492267    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492314    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492342    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492401    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492451    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.492612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.492578    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9545\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.530694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.530663    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:29.553942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.553917    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:08:29.556570 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:29.556519    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7288e662_5c7b_4c14_8379_f3bac4ff1c2c.slice/crio-f8f8f824c132efe8fc14960181d37942c98adb3eb2e854b9ddbda523f8f95fb0 WatchSource:0}: Error finding container f8f8f824c132efe8fc14960181d37942c98adb3eb2e854b9ddbda523f8f95fb0: Status 404 returned error can't find the container with id f8f8f824c132efe8fc14960181d37942c98adb3eb2e854b9ddbda523f8f95fb0
Nov 02 22:08:29.558877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.558841    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:29.558990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.558930    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:29.558990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.558978    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:29.593482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.593354    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.593657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.593499    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.593776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.593753    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594156    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594619    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594665    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594708    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594743    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.594846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.594809    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9545\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.595860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.595716    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.595987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.595969    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.596641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.596617    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.597046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.596983    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.597632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.597280    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.599856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.599757    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.601052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.601031    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.607086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.607059    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9545\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.607385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.607367    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.706626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.706306    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:29.819433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.819394    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:08:29.821332 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:29.821303    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1c6fdc_9bb4_43eb_a68b_5f1cc34ebec3.slice/crio-d7960f4e8f3bfc79c006477bb9bfcc1cf14a2962d365e77bbe2eb48d15306d45 WatchSource:0}: Error finding container d7960f4e8f3bfc79c006477bb9bfcc1cf14a2962d365e77bbe2eb48d15306d45: Status 404 returned error can't find the container with id d7960f4e8f3bfc79c006477bb9bfcc1cf14a2962d365e77bbe2eb48d15306d45
Nov 02 22:08:29.823841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.823807    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:29.823922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.823875    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:29.823922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:29.823904    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:30.028393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.028361    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:08:30.030977 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:30.030951    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1560d752_71a3_4287_95e4_a6cf2a172c12.slice/crio-c327abbcafca62afefdc1251cf0dc02acbcc7ac0517a5827e0afa060bb4a0d60 WatchSource:0}: Error finding container c327abbcafca62afefdc1251cf0dc02acbcc7ac0517a5827e0afa060bb4a0d60: Status 404 returned error can't find the container with id c327abbcafca62afefdc1251cf0dc02acbcc7ac0517a5827e0afa060bb4a0d60
Nov 02 22:08:30.033592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.033554    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:30.033714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.033649    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:30.033714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.033701    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:30.519925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.519885    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerStarted","Data":"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"}
Nov 02 22:08:30.520129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.519947    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerStarted","Data":"d7960f4e8f3bfc79c006477bb9bfcc1cf14a2962d365e77bbe2eb48d15306d45"}
Nov 02 22:08:30.521543 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.521503    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerStarted","Data":"04b9f3725e23b2149b465e6c19a22957de60a4cf563264d8c201341dc19cca5e"}
Nov 02 22:08:30.521678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.521569    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerStarted","Data":"f8f8f824c132efe8fc14960181d37942c98adb3eb2e854b9ddbda523f8f95fb0"}
Nov 02 22:08:30.523086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.523059    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerStarted","Data":"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"}
Nov 02 22:08:30.523201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.523101    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerStarted","Data":"c327abbcafca62afefdc1251cf0dc02acbcc7ac0517a5827e0afa060bb4a0d60"}
Nov 02 22:08:30.531556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.531512    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:30.537057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.537036    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:30.539782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.539733    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podStartSLOduration=1.539717585 podStartE2EDuration="1.539717585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.538313465 +0000 UTC m=+647.846404273" watchObservedRunningTime="2026-11-02 22:08:30.539717585 +0000 UTC m=+647.847808391"
Nov 02 22:08:30.575334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.575274    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podStartSLOduration=1.5752605160000002 podStartE2EDuration="1.575260516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.572699029 +0000 UTC m=+647.880789834" watchObservedRunningTime="2026-11-02 22:08:30.575260516 +0000 UTC m=+647.883351320"
Nov 02 22:08:30.590212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.590146    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podStartSLOduration=2.590129577 podStartE2EDuration="2.590129577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.589335451 +0000 UTC m=+647.897426256" watchObservedRunningTime="2026-11-02 22:08:30.590129577 +0000 UTC m=+647.898220382"
Nov 02 22:08:30.727729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:30.718653    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:31.263738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.263703    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:31.297692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.297657    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:31.530405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.530316    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:31.530405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.530383    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:31.531087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.531063    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:31.531189 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.531137    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:31.712445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:31.712412    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:32.534065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:32.534030    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:32.534724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:32.534706    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:08:33.733389 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.733339    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:08:33.737212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.737196    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.750376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.750353    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a1d78c\""
Nov 02 22:08:33.752398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.752363    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:08:33.805720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.805682    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:08:33.809285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.809267    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.819739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.819706    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a1d78c-dockercfg-jr979\""
Nov 02 22:08:33.821559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.821507    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:08:33.849705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.849653    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.849862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.849813    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.849862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.849853    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5cd\" (UniqueName: \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.850020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.849998    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfqp\" (UniqueName: \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.950882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.950844    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfqp\" (UniqueName: \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.950882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.950891    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.951138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.950939    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.951138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.950958    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5cd\" (UniqueName: \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.951739    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.951998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.951978    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:33.961861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.961835    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5cd\" (UniqueName: \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\") pod \"payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:33.961989 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:33.961967    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfqp\" (UniqueName: \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\") pod \"payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:34.050472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.050391    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:34.122459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.122392    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:34.322115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.322088    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:08:34.324250 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:34.324221    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53db915_22df_430d_9c2a_257a68c889a0.slice/crio-67e3415b38d5ae24fefe49201a80066a3b19cd71c6dc1cced8be17e7070bee3f WatchSource:0}: Error finding container 67e3415b38d5ae24fefe49201a80066a3b19cd71c6dc1cced8be17e7070bee3f: Status 404 returned error can't find the container with id 67e3415b38d5ae24fefe49201a80066a3b19cd71c6dc1cced8be17e7070bee3f
Nov 02 22:08:34.418309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.418279    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:08:34.420075 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:34.420043    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b0b6c3_a2c2_4c96_b223_4b3dd8aee51c.slice/crio-a4daedc8aad0a68b22aea861d70a0ba084ee265a615cbde4edecfde81bc45482 WatchSource:0}: Error finding container a4daedc8aad0a68b22aea861d70a0ba084ee265a615cbde4edecfde81bc45482: Status 404 returned error can't find the container with id a4daedc8aad0a68b22aea861d70a0ba084ee265a615cbde4edecfde81bc45482
Nov 02 22:08:34.466891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.466865    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:08:34.471952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.471928    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.485414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.484713    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-719012-dockercfg-8bzbm\""
Nov 02 22:08:34.486734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.486708    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:08:34.545636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.545583    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" event={"ID":"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c","Type":"ContainerStarted","Data":"ff85df32a437a53e6c13768b95052a204e45bb140105090532e9d6b19298da0d"}
Nov 02 22:08:34.545820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.545656    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" event={"ID":"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c","Type":"ContainerStarted","Data":"a4daedc8aad0a68b22aea861d70a0ba084ee265a615cbde4edecfde81bc45482"}
Nov 02 22:08:34.545820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.545771    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:34.551040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.551003    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" event={"ID":"d53db915-22df-430d-9c2a-257a68c889a0","Type":"ContainerStarted","Data":"4e55ee743b68e1567afbc5c4b413e9ecb3f0736128fdd58fc030eacf0eddbe5e"}
Nov 02 22:08:34.551201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.551108    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" event={"ID":"d53db915-22df-430d-9c2a-257a68c889a0","Type":"ContainerStarted","Data":"67e3415b38d5ae24fefe49201a80066a3b19cd71c6dc1cced8be17e7070bee3f"}
Nov 02 22:08:34.551201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.551146    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:34.555620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.555594    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-719012\""
Nov 02 22:08:34.558414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.558388    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgv7\" (UniqueName: \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.558670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.558643    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.564345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.564282    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" podStartSLOduration=1.5642648129999999 podStartE2EDuration="1.564264813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:34.560561275 +0000 UTC m=+651.868652075" watchObservedRunningTime="2026-11-02 22:08:34.564264813 +0000 UTC m=+651.872355670"
Nov 02 22:08:34.585384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.585259    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" podStartSLOduration=1.585239512 podStartE2EDuration="1.585239512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:34.584201693 +0000 UTC m=+651.892292500" watchObservedRunningTime="2026-11-02 22:08:34.585239512 +0000 UTC m=+651.893330320"
Nov 02 22:08:34.659601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.659553    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgv7\" (UniqueName: \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.659794 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.659653    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.660844 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.660816    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.667198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.667178    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgv7\" (UniqueName: \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\") pod \"payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.787931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.787897    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:34.988333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.988303    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:08:34.993088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:34.993071    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.000343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.000308    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f4a2f\""
Nov 02 22:08:35.001252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.001229    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:08:35.064125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.064095    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.064300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.064223    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwxk\" (UniqueName: \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.071967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.071647    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:08:35.079568 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:35.075020    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5776cdc7_9c34_4d5a_b6ab_6cfc19a55062.slice/crio-6c5b0933d4c34d494286766fad6b9ebd91e8263d35f4cdf7c1198a49ec0e9a30 WatchSource:0}: Error finding container 6c5b0933d4c34d494286766fad6b9ebd91e8263d35f4cdf7c1198a49ec0e9a30: Status 404 returned error can't find the container with id 6c5b0933d4c34d494286766fad6b9ebd91e8263d35f4cdf7c1198a49ec0e9a30
Nov 02 22:08:35.165815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.165786    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwxk\" (UniqueName: \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.165968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.165862    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.166856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.166837    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.174753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.174730    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwxk\" (UniqueName: \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\") pod \"payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.311821 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.311783    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:35.556726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.556688    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" event={"ID":"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062","Type":"ContainerStarted","Data":"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"}
Nov 02 22:08:35.556871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.556755    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" event={"ID":"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062","Type":"ContainerStarted","Data":"6c5b0933d4c34d494286766fad6b9ebd91e8263d35f4cdf7c1198a49ec0e9a30"}
Nov 02 22:08:35.574335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.574281    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" podStartSLOduration=1.574266929 podStartE2EDuration="1.574266929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:35.572806727 +0000 UTC m=+652.880897541" watchObservedRunningTime="2026-11-02 22:08:35.574266929 +0000 UTC m=+652.882357733"
Nov 02 22:08:35.589144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:35.589114    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:08:35.592411 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:35.592380    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35533e2b_61e1_43eb_9c57_cea4dfeea71f.slice/crio-29e5af6f64231016c3d06761818aa4eb450d6fc893a609410d91ae3e6f7044a8 WatchSource:0}: Error finding container 29e5af6f64231016c3d06761818aa4eb450d6fc893a609410d91ae3e6f7044a8: Status 404 returned error can't find the container with id 29e5af6f64231016c3d06761818aa4eb450d6fc893a609410d91ae3e6f7044a8
Nov 02 22:08:36.047518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.047418    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:08:36.052085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.052060    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.058898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.058873    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-719012-serving-cert\""
Nov 02 22:08:36.062167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.062055    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:08:36.197913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.197878    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.197913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.197916    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dsz\" (UniqueName: \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.198125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.198061    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.198125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.198097    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.299436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.299343    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.299436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.299383    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.299682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.299446    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.299682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.299472    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dsz\" (UniqueName: \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.300198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.300171    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.302079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.302056    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.302204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.302164    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.313497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.313467    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dsz\" (UniqueName: \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\") pod \"maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.376685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.376648    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:36.563454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.563369    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" event={"ID":"35533e2b-61e1-43eb-9c57-cea4dfeea71f","Type":"ContainerStarted","Data":"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"}
Nov 02 22:08:36.563454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.563422    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" event={"ID":"35533e2b-61e1-43eb-9c57-cea4dfeea71f","Type":"ContainerStarted","Data":"29e5af6f64231016c3d06761818aa4eb450d6fc893a609410d91ae3e6f7044a8"}
Nov 02 22:08:36.563677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.563463    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:36.563677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.563618    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:36.579761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.579714    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" podStartSLOduration=2.579700543 podStartE2EDuration="2.579700543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:36.577849255 +0000 UTC m=+653.885940070" watchObservedRunningTime="2026-11-02 22:08:36.579700543 +0000 UTC m=+653.887791368"
Nov 02 22:08:36.655448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.655415    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:08:36.660546 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:36.660511    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb363f6_1190_4f04_8e86_1695bd76d7ee.slice/crio-b635116e1bdc169449748b3671089d5bdbe164cded331b5c76c5a6b7a2c7b988 WatchSource:0}: Error finding container b635116e1bdc169449748b3671089d5bdbe164cded331b5c76c5a6b7a2c7b988: Status 404 returned error can't find the container with id b635116e1bdc169449748b3671089d5bdbe164cded331b5c76c5a6b7a2c7b988
Nov 02 22:08:36.662115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.662093    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:08:36.734789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.734754    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:08:36.739678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.739654    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:36.747980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.747960    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:08:36.748226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.748203    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-310bb3\""
Nov 02 22:08:36.906558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.906230    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:36.906558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:36.906390    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhnh\" (UniqueName: \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.007868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.007833    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.008034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.007890    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhnh\" (UniqueName: \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.008721 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.008696    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.017222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.017192    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhnh\" (UniqueName: \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\") pod \"payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.054677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.054647    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.329478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.329452    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:08:37.332216 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:37.332188    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35874e_a66a_4509_8705_6f359c931155.slice/crio-e0110da6a48b69f8f159c4ac88997d78634e5758ea13658d883b6650ef66ea79 WatchSource:0}: Error finding container e0110da6a48b69f8f159c4ac88997d78634e5758ea13658d883b6650ef66ea79: Status 404 returned error can't find the container with id e0110da6a48b69f8f159c4ac88997d78634e5758ea13658d883b6650ef66ea79
Nov 02 22:08:37.529326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.529253    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:08:37.533188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.533162    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.538454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.538429    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c302b6-dockercfg-s9qhl\""
Nov 02 22:08:37.543982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.543953    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:08:37.555880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.555834    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c302b6\""
Nov 02 22:08:37.571612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.571566    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" event={"ID":"5d35874e-a66a-4509-8705-6f359c931155","Type":"ContainerStarted","Data":"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"}
Nov 02 22:08:37.571765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.571634    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" event={"ID":"5d35874e-a66a-4509-8705-6f359c931155","Type":"ContainerStarted","Data":"e0110da6a48b69f8f159c4ac88997d78634e5758ea13658d883b6650ef66ea79"}
Nov 02 22:08:37.571860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.571796    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:37.573453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.573295    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" event={"ID":"0eb363f6-1190-4f04-8e86-1695bd76d7ee","Type":"ContainerStarted","Data":"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"}
Nov 02 22:08:37.573453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.573366    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" event={"ID":"0eb363f6-1190-4f04-8e86-1695bd76d7ee","Type":"ContainerStarted","Data":"b635116e1bdc169449748b3671089d5bdbe164cded331b5c76c5a6b7a2c7b988"}
Nov 02 22:08:37.573453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.573419    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:37.588867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.588813    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" podStartSLOduration=1.5887976 podStartE2EDuration="1.5887976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.586809084 +0000 UTC m=+654.894899889" watchObservedRunningTime="2026-11-02 22:08:37.5887976 +0000 UTC m=+654.896888406"
Nov 02 22:08:37.603005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.602960    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" podStartSLOduration=1.086411585 podStartE2EDuration="1.602947298s" totalImagesPullingTime="516.535713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.602920641 +0000 UTC m=+654.911011447" watchObservedRunningTime="2026-11-02 22:08:37.602947298 +0000 UTC m=+654.911038102"
Nov 02 22:08:37.614503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.614470    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.614636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.614517    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw44\" (UniqueName: \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.722922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.716974    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.722922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.717044    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw44\" (UniqueName: \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.722922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.718513    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.767299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.766288    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw44\" (UniqueName: \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\") pod \"payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.850909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.850878    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:37.977323 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:37.977286    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:08:38.155715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:38.155630    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:08:38.181609 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:38.181559    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3637ea_9eed_432f_a7b5_0c737b3d9055.slice/crio-b4b85cce71f2cec5f07f586e9c24e55aae767b66ae6449aa8ec8be9c193e9140 WatchSource:0}: Error finding container b4b85cce71f2cec5f07f586e9c24e55aae767b66ae6449aa8ec8be9c193e9140: Status 404 returned error can't find the container with id b4b85cce71f2cec5f07f586e9c24e55aae767b66ae6449aa8ec8be9c193e9140
Nov 02 22:08:38.592611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:38.592463    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" event={"ID":"5a3637ea-9eed-432f-a7b5-0c737b3d9055","Type":"ContainerStarted","Data":"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"}
Nov 02 22:08:38.592611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:38.592522    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" event={"ID":"5a3637ea-9eed-432f-a7b5-0c737b3d9055","Type":"ContainerStarted","Data":"b4b85cce71f2cec5f07f586e9c24e55aae767b66ae6449aa8ec8be9c193e9140"}
Nov 02 22:08:38.598673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:38.593125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:38.614416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:38.614354    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" podStartSLOduration=1.614336362 podStartE2EDuration="1.614336362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:38.613940602 +0000 UTC m=+655.922031406" watchObservedRunningTime="2026-11-02 22:08:38.614336362 +0000 UTC m=+655.922427169"
Nov 02 22:08:39.647733 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.647695    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:08:39.651570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.651528    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.658082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.658054    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-361a44-serving-cert\""
Nov 02 22:08:39.663308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.662840    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:08:39.741765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.741725    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.741915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.741844    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.741994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.741981    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvxk\" (UniqueName: \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.742065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.742045    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.843044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.843003    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.843227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.843069    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.843227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.843111    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.843227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.843141    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvxk\" (UniqueName: \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.843414 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:08:39.843252    4929 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-serving-cert: secret "maas-api-e2e-worker-w5-361a44-serving-cert" not found
Nov 02 22:08:39.843414 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:08:39.843327    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls podName:2a2fc86c-dde3-49c5-aa03-600f697954fd nodeName:}" failed. No retries permitted until 2026-11-02 22:08:40.343306484 +0000 UTC m=+657.651397281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls") pod "maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" (UID: "2a2fc86c-dde3-49c5-aa03-600f697954fd") : secret "maas-api-e2e-worker-w5-361a44-serving-cert" not found
Nov 02 22:08:39.843937 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.843914    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.845729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.845706    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:39.851433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:39.851413    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvxk\" (UniqueName: \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:40.358453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:40.358415    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:40.360840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:40.360815    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:40.567784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:40.567747    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:40.843862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:40.843837    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:08:40.845479 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:40.845444    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2fc86c_dde3_49c5_aa03_600f697954fd.slice/crio-df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7 WatchSource:0}: Error finding container df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7: Status 404 returned error can't find the container with id df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7
Nov 02 22:08:41.230776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.228547    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:08:41.233808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.233782    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.241764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.241648    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-310bb3-serving-cert\""
Nov 02 22:08:41.248837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.244719    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:08:41.370131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.370092    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.370286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.370210    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.370286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.370245    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fdr\" (UniqueName: \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.370412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.370348    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.472089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.472005    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.472225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.472120    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.472225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.472150    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fdr\" (UniqueName: \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.472225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.472202    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.472834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.472804    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.474718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.474691    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.474864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.474843    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.479976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.479949    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fdr\" (UniqueName: \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\") pod \"maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.572978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.572938    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:41.613426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.613386    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" event={"ID":"2a2fc86c-dde3-49c5-aa03-600f697954fd","Type":"ContainerStarted","Data":"7974a1565153e052385df152e8b689569333e24d5b4d1bd8d8da1d1ffaf8a6b2"}
Nov 02 22:08:41.613613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.613487    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:41.613613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.613514    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" event={"ID":"2a2fc86c-dde3-49c5-aa03-600f697954fd","Type":"ContainerStarted","Data":"df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7"}
Nov 02 22:08:41.876414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.876332    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" podStartSLOduration=2.509012476 podStartE2EDuration="2.876308239s" totalImagesPullingTime="367.295763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:41.630794387 +0000 UTC m=+658.938885190" watchObservedRunningTime="2026-11-02 22:08:41.876308239 +0000 UTC m=+659.184399044"
Nov 02 22:08:41.878911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:41.876962    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:08:41.885469 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:08:41.885349    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8cb3ee_6d94_4ec9_91e8_0f8284e99c4c.slice/crio-9968299f91dc133eb0d6b62eb17b02a6b275bbbce02e27a161c753e516218817 WatchSource:0}: Error finding container 9968299f91dc133eb0d6b62eb17b02a6b275bbbce02e27a161c753e516218817: Status 404 returned error can't find the container with id 9968299f91dc133eb0d6b62eb17b02a6b275bbbce02e27a161c753e516218817
Nov 02 22:08:42.620085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:42.620050    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" event={"ID":"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c","Type":"ContainerStarted","Data":"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"}
Nov 02 22:08:42.620290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:42.620110    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" event={"ID":"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c","Type":"ContainerStarted","Data":"9968299f91dc133eb0d6b62eb17b02a6b275bbbce02e27a161c753e516218817"}
Nov 02 22:08:42.620339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:42.620298    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:42.640842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:42.640778    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" podStartSLOduration=1.23099465 podStartE2EDuration="1.640764813s" totalImagesPullingTime="409.770163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:42.63911078 +0000 UTC m=+659.947201586" watchObservedRunningTime="2026-11-02 22:08:42.640764813 +0000 UTC m=+659.948855617"
Nov 02 22:08:42.893642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:42.893558    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:08:43.599217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:43.599188    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:08:45.586118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:45.586084    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:08:45.586486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:45.586177    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:08:47.588137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:47.588096    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:08:47.588692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:47.588177    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:08:47.629595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:47.629556    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:08:48.314423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:48.314389    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:08:48.594351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:48.594275    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:08:48.632956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:48.631572    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:08:50.611617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:50.611581    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:08:54.777102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.777070    4929 generic.go:356] "Generic (PLEG): container finished" podID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerID="7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe" exitCode=0
Nov 02 22:08:54.777407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.777155    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerDied","Data":"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"}
Nov 02 22:08:54.777708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.777687    4929 scope.go:122] "RemoveContainer" containerID="7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"
Nov 02 22:08:54.780133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.780089    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:54.780238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.780211    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:54.780401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:54.780362    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:55.784471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.784379    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerStarted","Data":"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"}
Nov 02 22:08:55.786183 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.786151    4929 generic.go:356] "Generic (PLEG): container finished" podID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerID="04b9f3725e23b2149b465e6c19a22957de60a4cf563264d8c201341dc19cca5e" exitCode=0
Nov 02 22:08:55.786345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.786212    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerDied","Data":"04b9f3725e23b2149b465e6c19a22957de60a4cf563264d8c201341dc19cca5e"}
Nov 02 22:08:55.786593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.786574    4929 scope.go:122] "RemoveContainer" containerID="04b9f3725e23b2149b465e6c19a22957de60a4cf563264d8c201341dc19cca5e"
Nov 02 22:08:55.787797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.787766    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:55.787888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.787832    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:55.787888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:55.787863    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:08:56.530926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:56.530884    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:08:56.531137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:56.531110    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 22:08:56.531216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:56.531168    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 22:08:56.815933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:56.815855    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerStarted","Data":"25080e7f7b4fefafc8629ab803963c2d803c5a6e25670b766f6bddfdb2fdf16e"}
Nov 02 22:08:57.268558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.267614    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:08:57.268558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.267764    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:08:57.268558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.267825    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:08:57.525630 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:08:57.521924    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:08:57.531332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.531300    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 22:08:57.531492 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.531361    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 22:08:57.608735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:57.608697    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:08:58.265981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:58.265941    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:08:58.266589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:58.266554    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:08:58.531631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:58.531423    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 22:08:58.531631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:58.531477    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 22:08:59.265429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:59.264958    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:08:59.265429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:59.265020    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:08:59.531214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:59.531133    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 22:08:59.531214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:08:59.531181    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 22:09:00.264558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:00.263961    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:09:00.264558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:00.264029    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:09:00.531286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:00.531217    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 22:09:00.531286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:00.531264    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 22:09:01.092128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.091644    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:09:01.107678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.105722    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.114998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.113999    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"kube-root-ca.crt\""
Nov 02 22:09:01.115280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.115005    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"openshift-service-ca.crt\""
Nov 02 22:09:01.115477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.115400    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:09:01.118025 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.118003    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"default-dockercfg-wgtlm\""
Nov 02 22:09:01.119798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.118611    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"simulator-w2-2532af-kserve-self-signed-certs\""
Nov 02 22:09:01.169284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.169229    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.169622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.169599    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.169789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.169769    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86wf\" (UniqueName: \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.170044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.170025    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.170168 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.170154    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.170274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.170261    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.273450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.273412    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:09:01.292982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.274321    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:09:01.293609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.273862    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.293846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.293824    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86wf\" (UniqueName: \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.294034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.294020    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.294135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.294124    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.294231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.294219    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.294367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.294356    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.294886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.294864    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.295031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.284016    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.295618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.295598    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.297082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.297060    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.301801 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.301774    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.313412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.313368    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86wf\" (UniqueName: \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\") pod \"simulator-w2-2532af-kserve-7f98c68b76-8nxxh\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") " pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.430392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.430356    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:01.575850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.574649    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:09:01.575850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.575838    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:09:01.586843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.586813    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:09:01.762353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.762274    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:09:01.766313 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:09:01.766287    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfae75d80_8497_4240_80d7_7b55d7b97b37.slice/crio-ca077dfdb94c7f2f1f68f6785b558cf1d0bf02221d309f971d391f971d71c2a2 WatchSource:0}: Error finding container ca077dfdb94c7f2f1f68f6785b558cf1d0bf02221d309f971d391f971d71c2a2: Status 404 returned error can't find the container with id ca077dfdb94c7f2f1f68f6785b558cf1d0bf02221d309f971d391f971d71c2a2
Nov 02 22:09:01.863978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.863938    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" event={"ID":"fae75d80-8497-4240-80d7-7b55d7b97b37","Type":"ContainerStarted","Data":"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"}
Nov 02 22:09:01.864194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.863992    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" event={"ID":"fae75d80-8497-4240-80d7-7b55d7b97b37","Type":"ContainerStarted","Data":"ca077dfdb94c7f2f1f68f6785b558cf1d0bf02221d309f971d391f971d71c2a2"}
Nov 02 22:09:01.865966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.865939    4929 generic.go:356] "Generic (PLEG): container finished" podID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerID="2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803" exitCode=0
Nov 02 22:09:01.866258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.866025    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerDied","Data":"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"}
Nov 02 22:09:01.866480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.866462    4929 scope.go:122] "RemoveContainer" containerID="2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"
Nov 02 22:09:01.867748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.867728    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:09:01.868186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.868152    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:09:01.868264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.868248    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:09:01.868324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.868285    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:09:01.881872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:01.881444    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podStartSLOduration=0.881430895 podStartE2EDuration="881.430895ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:01.879197523 +0000 UTC m=+679.187288338" watchObservedRunningTime="2026-11-02 22:09:01.881430895 +0000 UTC m=+679.189521700"
Nov 02 22:09:02.263336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:02.263303    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 22:09:02.263509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:02.263362    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 22:09:02.430824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:02.430784    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:02.534306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:02.534227    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:09:02.873092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:02.873058    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerStarted","Data":"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"}
Nov 02 22:09:03.268658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:03.268588    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:09:03.269645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:03.269617    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:09:03.711470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:03.709640    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:09:03.716453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:03.715914    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:03.716453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:03.715965    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 22:09:04.707850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:04.707807    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:04.708260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:04.707885    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 22:09:05.707327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:05.707281    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:05.707485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:05.707359    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 22:09:06.715813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:06.709285    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:06.715813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:06.709372    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 22:09:07.707262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:07.707225    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:07.707453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:07.707282    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 22:09:08.711911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:08.711877    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:09:08.712398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:08.712279    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:09:08.712944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:08.712924    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:09:09.908654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.908621    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:09:09.913641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.913613    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:09.920564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.920517    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:09:09.923021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.922999    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"kube-root-ca.crt\""
Nov 02 22:09:09.924313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.924291    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"openshift-service-ca.crt\""
Nov 02 22:09:09.924423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.924321    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"simulator-w4-c302b6-kserve-self-signed-certs\""
Nov 02 22:09:09.924423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:09.924405    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"default-dockercfg-p55xm\""
Nov 02 22:09:10.036372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036341    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.036572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036407    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgzgw\" (UniqueName: \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.036572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036460    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.036572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036502    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.036731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036710    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.036792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.036746    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138203    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138239    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138305    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138323    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgzgw\" (UniqueName: \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138371    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138421    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138700    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138721    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.138802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.138774    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.140663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.140637    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.140939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.140919    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.146335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.146312    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgzgw\" (UniqueName: \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\") pod \"simulator-w4-c302b6-kserve-58c59585cf-jsrdb\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") " pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.228453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.228371    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:10.504511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.504480    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:09:10.506676 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:09:10.506645    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod284396bb_975e_4e74_bb8c_e928d90b67a9.slice/crio-7a02e1dcd3dcf7366412a75cd5e3a153fa6d0e3ca87493b3906f259362f25fbf WatchSource:0}: Error finding container 7a02e1dcd3dcf7366412a75cd5e3a153fa6d0e3ca87493b3906f259362f25fbf: Status 404 returned error can't find the container with id 7a02e1dcd3dcf7366412a75cd5e3a153fa6d0e3ca87493b3906f259362f25fbf
Nov 02 22:09:10.937793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.937750    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" event={"ID":"284396bb-975e-4e74-bb8c-e928d90b67a9","Type":"ContainerStarted","Data":"3576a6e2f6c957740ce8ff2acf52996b8e0d99b994ccb9e86cf3f52080467c4b"}
Nov 02 22:09:10.938211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.937802    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" event={"ID":"284396bb-975e-4e74-bb8c-e928d90b67a9","Type":"ContainerStarted","Data":"7a02e1dcd3dcf7366412a75cd5e3a153fa6d0e3ca87493b3906f259362f25fbf"}
Nov 02 22:09:10.953816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:10.953768    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podStartSLOduration=1.953751993 podStartE2EDuration="1.953751993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:10.952379819 +0000 UTC m=+688.260470623" watchObservedRunningTime="2026-11-02 22:09:10.953751993 +0000 UTC m=+688.261842800"
Nov 02 22:09:11.229601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:11.229479    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:11.430598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:11.430562    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:11.442918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:11.442892    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:11.954102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:11.954072    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:09:12.564104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:12.564068    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:09:20.228796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:20.228744    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:20.241782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:20.241750    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:21.007969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:21.007935    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:09:23.870843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.870806    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:09:23.876173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.876147    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.885210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.885182    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:09:23.886228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.886202    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"kube-root-ca.crt\""
Nov 02 22:09:23.886362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.886252    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"default-dockercfg-7s9mk\""
Nov 02 22:09:23.886362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.886297    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"openshift-service-ca.crt\""
Nov 02 22:09:23.887624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.887446    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"premium-w1-5f4a2f-kserve-self-signed-certs\""
Nov 02 22:09:23.998653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998617    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.998835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998677    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.998835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998762    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.998835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998791    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.998964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998838    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:23.998964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:23.998894    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb2v\" (UniqueName: \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113094    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113416    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113480    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113522    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113607    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.113802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.113657    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb2v\" (UniqueName: \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.114513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.114475    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.115112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.115082    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.115269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.115199    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.127926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.122877    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.148606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.148570    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.152498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.151723    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb2v\" (UniqueName: \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\") pod \"premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") " pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.194266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.194226    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:24.469098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:24.469067    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:09:24.470690 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:09:24.470664    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfe65aff_64bc_4387_a4ae_46f4064ad0ce.slice/crio-67449cd295a1fd22622e7458b74dd1b1b8d08bfe3cab17955ac6355eb18469a3 WatchSource:0}: Error finding container 67449cd295a1fd22622e7458b74dd1b1b8d08bfe3cab17955ac6355eb18469a3: Status 404 returned error can't find the container with id 67449cd295a1fd22622e7458b74dd1b1b8d08bfe3cab17955ac6355eb18469a3
Nov 02 22:09:25.020739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:25.020703    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" event={"ID":"bfe65aff-64bc-4387-a4ae-46f4064ad0ce","Type":"ContainerStarted","Data":"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"}
Nov 02 22:09:25.020739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:25.020749    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" event={"ID":"bfe65aff-64bc-4387-a4ae-46f4064ad0ce","Type":"ContainerStarted","Data":"67449cd295a1fd22622e7458b74dd1b1b8d08bfe3cab17955ac6355eb18469a3"}
Nov 02 22:09:25.037947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:25.037901    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podStartSLOduration=2.037887185 podStartE2EDuration="2.037887185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:25.03589851 +0000 UTC m=+702.343989315" watchObservedRunningTime="2026-11-02 22:09:25.037887185 +0000 UTC m=+702.345977990"
Nov 02 22:09:25.195089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:25.195061    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:27.496495 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:09:27.496449    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:09:28.580829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:28.580794    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:09:34.195027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:34.194993    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:34.208029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:34.208003    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:35.115987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:35.115958    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:09:37.449223 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.449190    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:09:37.454205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.454181    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.458940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.458911    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.459090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.459063    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.459177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.459154    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.459263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.459247    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmcr\" (UniqueName: \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.459346 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.459331    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.459415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.459400    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.462051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.462025    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:09:37.462329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.462297    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"default-dockercfg-cc2pz\""
Nov 02 22:09:37.462329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.462325    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"openshift-service-ca.crt\""
Nov 02 22:09:37.462516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.462375    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"premium-w0-310bb3-kserve-self-signed-certs\""
Nov 02 22:09:37.462516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.462469    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"kube-root-ca.crt\""
Nov 02 22:09:37.560150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560117    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560156    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560198    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmcr\" (UniqueName: \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560230    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560255    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560287    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560712    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560799    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.560866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.560806    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.562705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.562679    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.562954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.562933    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.583567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.577507    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmcr\" (UniqueName: \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\") pod \"premium-w0-310bb3-kserve-6886cf5989-b9xjr\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") " pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:37.769432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:37.769337    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:38.042564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:38.042519    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:09:38.045458 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:09:38.045427    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43a8b213_8821_404a_8f1b_a0cbbe32793b.slice/crio-cccb33285d884e95e21c65632886a1ca7b9bc4708ace168d3c59aad9bd8b9d2c WatchSource:0}: Error finding container cccb33285d884e95e21c65632886a1ca7b9bc4708ace168d3c59aad9bd8b9d2c: Status 404 returned error can't find the container with id cccb33285d884e95e21c65632886a1ca7b9bc4708ace168d3c59aad9bd8b9d2c
Nov 02 22:09:38.132060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:38.132025    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" event={"ID":"43a8b213-8821-404a-8f1b-a0cbbe32793b","Type":"ContainerStarted","Data":"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"}
Nov 02 22:09:38.132237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:38.132071    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" event={"ID":"43a8b213-8821-404a-8f1b-a0cbbe32793b","Type":"ContainerStarted","Data":"cccb33285d884e95e21c65632886a1ca7b9bc4708ace168d3c59aad9bd8b9d2c"}
Nov 02 22:09:38.148562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:38.148501    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podStartSLOduration=1.148485602 podStartE2EDuration="1.148485602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:38.146778934 +0000 UTC m=+715.454869741" watchObservedRunningTime="2026-11-02 22:09:38.148485602 +0000 UTC m=+715.456576470"
Nov 02 22:09:38.769746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:38.769709    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:46.183220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:46.183183    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:09:47.769880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:47.769845    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:47.783083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:47.783057    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:48.234981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:48.234949    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:09:57.495822 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:09:57.495786    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:09:59.414091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.414051    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:09:59.418036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.418017    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.425125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.425099    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1d78c-openshift-default-dockercfg-dr8jk\""
Nov 02 22:09:59.429147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.429119    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:09:59.545182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545142    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545214    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545328    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545438 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545383    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545438 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545412    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545438 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545433    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545661    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545729    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.545808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.545764    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcz5\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647292    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647373    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647398    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647423    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647455    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647586    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647723    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647782    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647816    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcz5\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647916    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.647962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.647939    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.648092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.648050    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.648203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.648182    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.648339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.648314    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.649837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.649819    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.650215 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.650198    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.655693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.655672    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.655792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.655770    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcz5\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:09:59.734423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:09:59.734328    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:10:00.013909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.013882    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:10:00.015367 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:10:00.015340    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35494137_6f30_43ae_a2e1_f0ba686330d8.slice/crio-1ffa6aca429f0cd439e7fc837cc6fa1566eb227c363417e50dca59da4ddcad90 WatchSource:0}: Error finding container 1ffa6aca429f0cd439e7fc837cc6fa1566eb227c363417e50dca59da4ddcad90: Status 404 returned error can't find the container with id 1ffa6aca429f0cd439e7fc837cc6fa1566eb227c363417e50dca59da4ddcad90
Nov 02 22:10:00.017980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.017941    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:00.018062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.018018    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:00.018062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.018050    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:00.304998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.298702    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" event={"ID":"35494137-6f30-43ae-a2e1-f0ba686330d8","Type":"ContainerStarted","Data":"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"}
Nov 02 22:10:00.318557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.309037    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" event={"ID":"35494137-6f30-43ae-a2e1-f0ba686330d8","Type":"ContainerStarted","Data":"1ffa6aca429f0cd439e7fc837cc6fa1566eb227c363417e50dca59da4ddcad90"}
Nov 02 22:10:00.334563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.331219    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podStartSLOduration=1.331199156 podStartE2EDuration="1.331199156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:00.330993375 +0000 UTC m=+737.639084183" watchObservedRunningTime="2026-11-02 22:10:00.331199156 +0000 UTC m=+737.639289965"
Nov 02 22:10:00.734674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:00.734628    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:10:01.749609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:01.749571    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 22:10:01.750260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:01.750226    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 22:10:02.748572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:02.747993    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 22:10:02.748572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:02.748057    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 22:10:03.741565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:03.740924    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 22:10:03.741565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:03.740990    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 22:10:04.752609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:04.737934    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 22:10:04.752609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:04.737999    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 22:10:05.759560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:05.757675    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 22:10:05.759560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:05.757745    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 22:10:06.993739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:06.993704    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:10:06.994259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:06.994225    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:10:06.995292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:06.995263    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:10:09.268717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:09.268682    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:10:14.588903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.588871    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:10:14.593743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.593725    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.601051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.601008    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c302b6-openshift-default-dockercfg-wc6gv\""
Nov 02 22:10:14.605471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.605444    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:10:14.623796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.623761    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624074    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624350    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624579    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624625    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624670    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmx9\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624738    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624790    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.624977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.624836    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726005    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726054    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726074    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726140    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726192    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726229    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726265    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmx9\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726327    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726378    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726467    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726512    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726630    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726828 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726811    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.726973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.726956    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.728663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.728636    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.728779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.728754    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.747970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.747943    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.748086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.748038    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmx9\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:14.910267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:14.910233    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:15.191041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.191010    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:10:15.192585 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:10:15.192550    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedfa64a_113a_4cdc_bcb6_5e84b818e8f9.slice/crio-e4e8353a262eb21500a0e5fe757c74eee4d334d7987de58005056891131ae72a WatchSource:0}: Error finding container e4e8353a262eb21500a0e5fe757c74eee4d334d7987de58005056891131ae72a: Status 404 returned error can't find the container with id e4e8353a262eb21500a0e5fe757c74eee4d334d7987de58005056891131ae72a
Nov 02 22:10:15.195066 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.195032    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:15.195146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.195100    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:15.195146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.195132    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:15.411175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.411136    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" event={"ID":"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9","Type":"ContainerStarted","Data":"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"}
Nov 02 22:10:15.411359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.411189    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" event={"ID":"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9","Type":"ContainerStarted","Data":"e4e8353a262eb21500a0e5fe757c74eee4d334d7987de58005056891131ae72a"}
Nov 02 22:10:15.431873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.431826    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podStartSLOduration=1.431813284 podStartE2EDuration="1.431813284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:15.429920025 +0000 UTC m=+752.738010831" watchObservedRunningTime="2026-11-02 22:10:15.431813284 +0000 UTC m=+752.739904089"
Nov 02 22:10:15.911047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:15.911006    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:16.912475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:16.912442    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 22:10:16.912911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:16.912497    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 22:10:17.911413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:17.911376    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 22:10:17.911620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:17.911441    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 22:10:18.910843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:18.910810    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 22:10:18.911283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:18.910863    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 22:10:19.910732 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:19.910694    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 22:10:19.910912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:19.910773    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 22:10:20.910626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:20.910588    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 22:10:20.910908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:20.910882    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 22:10:21.950873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:21.950838    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:21.951555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:21.951506    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:21.952838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:21.952797    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:10:27.498684 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:10:27.498599    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:10:35.783405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:35.783372    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:10:49.443939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.443890    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:10:49.449844 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.449822    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.455136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.455110    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"default-dockercfg-q9cxc\""
Nov 02 22:10:49.456035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.456006    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:10:49.458011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.457987    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"distinct-w3-719012-kserve-self-signed-certs\""
Nov 02 22:10:49.459527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.459508    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"openshift-service-ca.crt\""
Nov 02 22:10:49.459633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.459614    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"kube-root-ca.crt\""
Nov 02 22:10:49.502331 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502293    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.502331 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502336    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.502521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502412    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqvz\" (UniqueName: \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.502521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502493    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.502681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502527    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.502681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.502635    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605179    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605250    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605304    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605412    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605451    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.605556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.605493    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqvz\" (UniqueName: \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.612040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.606365    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.612040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.607483    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.612040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.607813    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.613992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.613960    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.620231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.618007    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.620231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.620096    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqvz\" (UniqueName: \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\") pod \"distinct-w3-719012-kserve-7f856fd7b8-7pqbs\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") " pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:49.764328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:49.764252    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:50.059028 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:50.059002    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:10:50.061264 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:10:50.061222    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f3404cc_af7a_4aba_b97a_a62738a90624.slice/crio-afed03889b5f8a4bc084304dc67943a7884cfc206011857ee6e679fd35f55f3c WatchSource:0}: Error finding container afed03889b5f8a4bc084304dc67943a7884cfc206011857ee6e679fd35f55f3c: Status 404 returned error can't find the container with id afed03889b5f8a4bc084304dc67943a7884cfc206011857ee6e679fd35f55f3c
Nov 02 22:10:50.667914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:50.667879    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" event={"ID":"2f3404cc-af7a-4aba-b97a-a62738a90624","Type":"ContainerStarted","Data":"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"}
Nov 02 22:10:50.667914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:50.667921    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" event={"ID":"2f3404cc-af7a-4aba-b97a-a62738a90624","Type":"ContainerStarted","Data":"afed03889b5f8a4bc084304dc67943a7884cfc206011857ee6e679fd35f55f3c"}
Nov 02 22:10:50.687588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:50.687519    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podStartSLOduration=1.68750723 podStartE2EDuration="1.68750723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:50.685768165 +0000 UTC m=+787.993858972" watchObservedRunningTime="2026-11-02 22:10:50.68750723 +0000 UTC m=+787.995598034"
Nov 02 22:10:50.765150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:50.765117    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:57.496894 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:10:57.496855    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:10:59.232327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.232289    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:10:59.236692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.236672    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.240667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.240573    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f4a2f-openshift-default-dockercfg-d97zb\""
Nov 02 22:10:59.245246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.245221    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:10:59.320768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.320732    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:10:59.324700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.324678    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.335399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.335370    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:10:59.343411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.341684    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-719012-openshift-default-dockercfg-glgls\""
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354030    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354089    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354143    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354307    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354454    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.354563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354496    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.354898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354595    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.354898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354880    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.354996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354923    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.354996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.354978    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.355087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355062    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wt9\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.355194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355177    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.355261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355206    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.355315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355275    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.355378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355356    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.355465 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355437    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.355638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355475    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.355638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.355502    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457012    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457062    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457103    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457147    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457177    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457199    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457221    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457264    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457296    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457360    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457398    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wt9\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457431    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457456    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.457642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457486    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.458127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457650    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.458127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.457918    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.458363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458323    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.458606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458586    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.458773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458742    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.458888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458774    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.458888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458756    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.458888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.458853    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.459249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.459193    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.459373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.459274    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.460000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.459565    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.460000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.459920    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.460214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.460056    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.460470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.460449    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.460704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.460685    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.460704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.460694    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.461464 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.461445    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.461962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.461943    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.465270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.465243    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.468360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.468339    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.472607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.472512    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.472690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.472609    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wt9\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.554684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.554596    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:10:59.657369 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.657090    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:10:59.764480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.764449    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:59.780959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.780930    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:10:59.839297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.839260    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:10:59.841304 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:10:59.841278    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915f1229_f294_438f_aeef_f3ee3d57a67c.slice/crio-ba7dafccf41478dfac9c31a2a109df6dea12b78e0da5df1e3c6d5c938e0de957 WatchSource:0}: Error finding container ba7dafccf41478dfac9c31a2a109df6dea12b78e0da5df1e3c6d5c938e0de957: Status 404 returned error can't find the container with id ba7dafccf41478dfac9c31a2a109df6dea12b78e0da5df1e3c6d5c938e0de957
Nov 02 22:10:59.844043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.844008    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:59.844146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.844095    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:59.844192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.844149    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:59.982354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.982322    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:10:59.984614 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:10:59.984586    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747b4185_405d_48f1_913c_fafe99cd3d82.slice/crio-4f062f1f9fad377c79c4eb38e0fb2b130cbd1b8018f99ea099f0a9dfaf646c64 WatchSource:0}: Error finding container 4f062f1f9fad377c79c4eb38e0fb2b130cbd1b8018f99ea099f0a9dfaf646c64: Status 404 returned error can't find the container with id 4f062f1f9fad377c79c4eb38e0fb2b130cbd1b8018f99ea099f0a9dfaf646c64
Nov 02 22:10:59.987697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.987659    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:59.987768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.987732    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:10:59.987809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:10:59.987794    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:00.785937 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.785895    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" event={"ID":"747b4185-405d-48f1-913c-fafe99cd3d82","Type":"ContainerStarted","Data":"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"}
Nov 02 22:11:00.786442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.785967    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" event={"ID":"747b4185-405d-48f1-913c-fafe99cd3d82","Type":"ContainerStarted","Data":"4f062f1f9fad377c79c4eb38e0fb2b130cbd1b8018f99ea099f0a9dfaf646c64"}
Nov 02 22:11:00.789659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.789621    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" event={"ID":"915f1229-f294-438f-aeef-f3ee3d57a67c","Type":"ContainerStarted","Data":"607b6ae88b9c6178191dd0f5001967b857d8795cf2864cecdb0334da4a6d444d"}
Nov 02 22:11:00.789818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.789681    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" event={"ID":"915f1229-f294-438f-aeef-f3ee3d57a67c","Type":"ContainerStarted","Data":"ba7dafccf41478dfac9c31a2a109df6dea12b78e0da5df1e3c6d5c938e0de957"}
Nov 02 22:11:00.806788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.806761    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:11:00.807008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.806919    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podStartSLOduration=1.806905539 podStartE2EDuration="1.806905539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:00.806213123 +0000 UTC m=+798.114303938" watchObservedRunningTime="2026-11-02 22:11:00.806905539 +0000 UTC m=+798.114996343"
Nov 02 22:11:00.824488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:00.824443    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podStartSLOduration=1.824429339 podStartE2EDuration="1.824429339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:00.822910907 +0000 UTC m=+798.131001714" watchObservedRunningTime="2026-11-02 22:11:00.824429339 +0000 UTC m=+798.132520143"
Nov 02 22:11:01.570653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.570608    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:11:01.574703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.574673    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 22:11:01.574959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.574921    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 22:11:01.666560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.665960    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:11:01.674557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.674163    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 22:11:01.674557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:01.674216    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 22:11:02.559486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:02.559026    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 22:11:02.559486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:02.559096    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 22:11:02.660558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:02.659990    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 22:11:02.660558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:02.660060    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 22:11:03.226713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.221647    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:11:03.233689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.233658    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:11:03.234070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.233783    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.294749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.294718    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"default-dockercfg-hxfz5\""
Nov 02 22:11:03.295129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.295108    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"kube-root-ca.crt\""
Nov 02 22:11:03.295358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.295341    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"distinct-w5-361a44-kserve-self-signed-certs\""
Nov 02 22:11:03.295617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.295600    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"openshift-service-ca.crt\""
Nov 02 22:11:03.434301 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.434257    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.434631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.434603    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.434815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.434795    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.435023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.435005    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.435147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.435133    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.438556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.435289    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm57l\" (UniqueName: \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555245    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555315    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555422    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555463    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555548    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm57l\" (UniqueName: \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.555619    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.556120    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.557608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.556753    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.573558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.567941    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.585000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.575035    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.585000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.584857    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.585940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.585650    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 22:11:03.585940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.585722    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 22:11:03.589201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.589176    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm57l\" (UniqueName: \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\") pod \"distinct-w5-361a44-kserve-865769cb79-jtpql\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.613183 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.613154    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:03.662328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.661868    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 22:11:03.662328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.661930    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 22:11:03.986997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:03.985602    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:11:04.557571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.556996    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 22:11:04.557571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.557062    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 22:11:04.659562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.657999    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 22:11:04.659562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.658064    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 22:11:04.815931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.815850    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" event={"ID":"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b","Type":"ContainerStarted","Data":"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"}
Nov 02 22:11:04.815931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.815890    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" event={"ID":"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b","Type":"ContainerStarted","Data":"dd8b869926deeea2dce8df2f25b007c57ad01af7a189167a46f1a3567508c3ac"}
Nov 02 22:11:04.832073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:04.832023    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podStartSLOduration=1.8320080920000001 podStartE2EDuration="1.832008092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:04.830607585 +0000 UTC m=+802.138698387" watchObservedRunningTime="2026-11-02 22:11:04.832008092 +0000 UTC m=+802.140098896"
Nov 02 22:11:05.422398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.422363    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:11:05.427902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.427867    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.464560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.458635    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:11:05.465795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.465770    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"distinct-2-w2-2532af-kserve-self-signed-certs\""
Nov 02 22:11:05.517641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517592    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.517819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517674    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskmc\" (UniqueName: \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.517819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517695    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.517819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517798    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.518027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517911    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.518027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.517937    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.573558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.572651    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 22:11:05.573558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.572714    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 22:11:05.618775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.613900    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:05.619040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619006    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.619119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619092    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskmc\" (UniqueName: \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.619187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619121    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.619243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619216    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.619345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619327    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.619421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.619357    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.620964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.620914    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.622024    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.622332    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.628766    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.631497    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskmc\" (UniqueName: \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.651640    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\") pod \"distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.660997    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 22:11:05.664994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.661057    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 22:11:05.795604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:05.795507    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:06.116757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.116723    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:11:06.117617 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:06.117585    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b413e10_d99a_487b_bdf5_371604909fe0.slice/crio-2bf72c4738b0e02ee1a03f3b9ae4206f23c6607a6a132074ef08b35c61e56fad WatchSource:0}: Error finding container 2bf72c4738b0e02ee1a03f3b9ae4206f23c6607a6a132074ef08b35c61e56fad: Status 404 returned error can't find the container with id 2bf72c4738b0e02ee1a03f3b9ae4206f23c6607a6a132074ef08b35c61e56fad
Nov 02 22:11:06.821387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.821336    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:11:06.821945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.821908    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:11:06.824237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.824208    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:11:06.833900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.833235    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" event={"ID":"1b413e10-d99a-487b-bdf5-371604909fe0","Type":"ContainerStarted","Data":"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"}
Nov 02 22:11:06.833900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.833283    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" event={"ID":"1b413e10-d99a-487b-bdf5-371604909fe0","Type":"ContainerStarted","Data":"2bf72c4738b0e02ee1a03f3b9ae4206f23c6607a6a132074ef08b35c61e56fad"}
Nov 02 22:11:06.844692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.844654    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:11:06.845587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.845026    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:11:06.846438 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.846416    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:11:06.859060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:06.859009    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podStartSLOduration=1.8589958420000001 podStartE2EDuration="1.858995842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:06.85733858 +0000 UTC m=+804.165429388" watchObservedRunningTime="2026-11-02 22:11:06.858995842 +0000 UTC m=+804.167086647"
Nov 02 22:11:07.796229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:07.796184    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:08.018879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:08.018847    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:11:13.613710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:13.613674    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:13.626664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:13.626630    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:13.941105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:13.941021    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:11:14.187746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.187709    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:11:14.191845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.191800    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.207773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.207705    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:11:14.282423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282381    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7t7\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282433    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282524    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282663    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282705    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282792    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282841    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282912    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.282957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.282940    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383632    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383675    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383728    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383765    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383812    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7t7\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.383855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383852    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383901    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.383978    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384026    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384208    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384252    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384407    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384515 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384496    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.384608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.384495    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.386155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.386132    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.386265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.386236    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.391353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.391327    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.391497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.391478    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7t7\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.509653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.509564    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:11:14.512038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.512010    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:14.514679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.514657    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.525079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.525027    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:11:14.587125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.587091    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.587338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.587318    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.587494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.587478    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.587791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.587768    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.587944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.587916    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6l2r\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.590049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.590005    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.593610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.590193    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.598735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.598626    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.598735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.598704    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699424    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699472    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6l2r\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699494    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699521    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699564    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699584    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699617    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699652    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699720    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.699977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699944    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.700311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.699978    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.700311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.700134    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.700311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.700194    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.700412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.700325    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.702035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.702011    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.702379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.702358    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.712973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.707507    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.712973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.707589    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6l2r\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.806202 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:14.806169    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2568beb3_4483_41c4_8f42_8fba1a39ea2b.slice/crio-217ebc732b864b87e2b5815ba65d36b9c4a42c309bfc06a431057b143b2efc47 WatchSource:0}: Error finding container 217ebc732b864b87e2b5815ba65d36b9c4a42c309bfc06a431057b143b2efc47: Status 404 returned error can't find the container with id 217ebc732b864b87e2b5815ba65d36b9c4a42c309bfc06a431057b143b2efc47
Nov 02 22:11:14.806581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.806561    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:11:14.808413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.808382    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:14.808496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.808443    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:14.808496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.808471    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:14.844869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.841970    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:11:14.847722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.847696    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.854089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.854065    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:11:14.855962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.855888    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"distinct-w1-5f4a2f-kserve-self-signed-certs\""
Nov 02 22:11:14.863090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.863060    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:14.903474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903122    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.903474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903163    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgdb\" (UniqueName: \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.903474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903186    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.903474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903431    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.903851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903670    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.903851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.903802    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:14.945284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.945251    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" event={"ID":"2568beb3-4483-41c4-8f42-8fba1a39ea2b","Type":"ContainerStarted","Data":"021c753bc0535e987794a02473be995813d5650863ab568f00602268f4824a06"}
Nov 02 22:11:14.945431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.945315    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" event={"ID":"2568beb3-4483-41c4-8f42-8fba1a39ea2b","Type":"ContainerStarted","Data":"217ebc732b864b87e2b5815ba65d36b9c4a42c309bfc06a431057b143b2efc47"}
Nov 02 22:11:14.966145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:14.966085    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podStartSLOduration=0.966065193 podStartE2EDuration="966.065193ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:14.96212359 +0000 UTC m=+812.270214395" watchObservedRunningTime="2026-11-02 22:11:14.966065193 +0000 UTC m=+812.274155998"
Nov 02 22:11:15.068788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.068700    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.105654    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.105728    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.105759    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgdb\" (UniqueName: \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.105794    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.105886    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.085920    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.107169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.106796    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.109612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.108371    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.112441 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.112396    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.117169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.117082    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.117393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.117370    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgdb\" (UniqueName: \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\") pod \"distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.151595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.151553    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:11:15.152045 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:15.152005    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdde8b0_b059_4ba1_a0ad_41de69c3974e.slice/crio-7372fae050648cf41878c4f32570033bc58926ae7947b13ca2455b156fb2ea5f WatchSource:0}: Error finding container 7372fae050648cf41878c4f32570033bc58926ae7947b13ca2455b156fb2ea5f: Status 404 returned error can't find the container with id 7372fae050648cf41878c4f32570033bc58926ae7947b13ca2455b156fb2ea5f
Nov 02 22:11:15.157214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.157170    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:15.157299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.157279    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:15.157348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.157332    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:15.176129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.176106    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:15.472647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.472619    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:11:15.474470 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:15.474441    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23fa1f86_dd73_4b71_8eeb_4229428a8649.slice/crio-d105e65892a48ba8f8b31509eceedfb028e990259462f072ac7e02dea63025d4 WatchSource:0}: Error finding container d105e65892a48ba8f8b31509eceedfb028e990259462f072ac7e02dea63025d4: Status 404 returned error can't find the container with id d105e65892a48ba8f8b31509eceedfb028e990259462f072ac7e02dea63025d4
Nov 02 22:11:15.513195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.513157    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:15.514586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.514560    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:15.514726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.514617    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:15.796483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.796396    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:15.816559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.816511    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:15.989171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.987907    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" event={"ID":"23fa1f86-dd73-4b71-8eeb-4229428a8649","Type":"ContainerStarted","Data":"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"}
Nov 02 22:11:15.989171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.987977    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" event={"ID":"23fa1f86-dd73-4b71-8eeb-4229428a8649","Type":"ContainerStarted","Data":"d105e65892a48ba8f8b31509eceedfb028e990259462f072ac7e02dea63025d4"}
Nov 02 22:11:15.991320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.991284    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" event={"ID":"4bdde8b0-b059-4ba1-a0ad-41de69c3974e","Type":"ContainerStarted","Data":"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"}
Nov 02 22:11:15.991461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:15.991348    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" event={"ID":"4bdde8b0-b059-4ba1-a0ad-41de69c3974e","Type":"ContainerStarted","Data":"7372fae050648cf41878c4f32570033bc58926ae7947b13ca2455b156fb2ea5f"}
Nov 02 22:11:16.003586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.003517    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:11:16.006092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.006044    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podStartSLOduration=2.006031537 podStartE2EDuration="2.006031537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:16.004312289 +0000 UTC m=+813.312403093" watchObservedRunningTime="2026-11-02 22:11:16.006031537 +0000 UTC m=+813.314122341"
Nov 02 22:11:16.059615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.056163    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podStartSLOduration=2.056144473 podStartE2EDuration="2.056144473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:16.02878713 +0000 UTC m=+813.336877936" watchObservedRunningTime="2026-11-02 22:11:16.056144473 +0000 UTC m=+813.364235279"
Nov 02 22:11:16.187676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.187638    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:16.525562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.517617    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:16.525562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.517692    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:16.879731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.879687    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:16.887581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.886935    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:16.887581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:16.887009    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:17.514991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:17.514949    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:17.515295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:17.515261    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:17.864010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:17.863971    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:17.864295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:17.864263    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:18.520029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:18.519986    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:18.520940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:18.520902    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:18.872558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:18.864936    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:18.872558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:18.865000    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:19.512606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:19.512566    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:19.512775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:19.512654    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:19.872586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:19.872526    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:19.873162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:19.873127    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:20.512810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:20.512774    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:20.513065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:20.512998    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:20.865559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:20.863974    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:20.865559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:20.864040    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:21.512947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:21.512907    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 22:11:21.513440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:21.512986    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 22:11:21.864044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:21.864002    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 22:11:21.864237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:21.864086    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 22:11:22.516981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.516951    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:22.517372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.517353    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:22.518188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.518172    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:11:22.868607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.868571    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:22.868984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.868957    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:22.869907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:22.869888    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:11:24.452124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.452083    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:11:24.458252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.458227    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.465054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.465025    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:11:24.468130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.468107    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"default-dockercfg-sz7zh\""
Nov 02 22:11:24.468256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.468129    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"openshift-service-ca.crt\""
Nov 02 22:11:24.468256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.468170    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"kube-root-ca.crt\""
Nov 02 22:11:24.469486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.469453    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"distinct-2-w6-a1d78c-kserve-self-signed-certs\""
Nov 02 22:11:24.474270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474246    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.474368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474279    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.474368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474302    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4qd\" (UniqueName: \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.474457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474424    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.474621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474592    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.474721 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.474646    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575738    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575788    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575858    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575885    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575902    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4qd\" (UniqueName: \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.575966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.575954    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.576290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.576266    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.576358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.576304    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.576416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.576372    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.578008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.577983    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.578581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.578527    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.583921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.583902    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4qd\" (UniqueName: \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\") pod \"distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:24.777513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:24.777427    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:25.059952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:25.059922    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:11:25.062022 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:25.061995    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb08deb0f_241c_478a_b608_d9409bc3c8d8.slice/crio-fdc77fe0d76e8dc9b9e0e20889d3f87c14194fabcb54af1ee7658e615aa0f57a WatchSource:0}: Error finding container fdc77fe0d76e8dc9b9e0e20889d3f87c14194fabcb54af1ee7658e615aa0f57a: Status 404 returned error can't find the container with id fdc77fe0d76e8dc9b9e0e20889d3f87c14194fabcb54af1ee7658e615aa0f57a
Nov 02 22:11:25.082979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:25.082943    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" event={"ID":"b08deb0f-241c-478a-b608-d9409bc3c8d8","Type":"ContainerStarted","Data":"fdc77fe0d76e8dc9b9e0e20889d3f87c14194fabcb54af1ee7658e615aa0f57a"}
Nov 02 22:11:25.176318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:25.176267    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:25.189857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:25.189827    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:26.093344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:26.093304    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" event={"ID":"b08deb0f-241c-478a-b608-d9409bc3c8d8","Type":"ContainerStarted","Data":"0bf45d1e3fcac438ef014afa02a8b0aa631f45289d93db7e723bef4d95a03edb"}
Nov 02 22:11:26.105230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:26.105202    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:11:26.112285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:26.112240    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podStartSLOduration=2.112227586 podStartE2EDuration="2.112227586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:26.109956779 +0000 UTC m=+823.418047585" watchObservedRunningTime="2026-11-02 22:11:26.112227586 +0000 UTC m=+823.420318390"
Nov 02 22:11:26.777790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:26.777748    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:27.702884 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:11:27.702830    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:11:34.778119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:34.778077    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:34.791259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:34.791230    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:35.166775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:35.166748    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:11:36.958887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:36.958849    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:11:42.760955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.760917    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:11:42.767359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.767332    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.772235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.772205    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:11:42.777657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.775334    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"distinct-2-w5-361a44-kserve-self-signed-certs\""
Nov 02 22:11:42.788118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788080    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.788262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788195    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxlj\" (UniqueName: \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.788329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788248    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.788383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788341    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.788383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788369    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.788490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.788392    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.922699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.904275    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxlj\" (UniqueName: \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.923024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.922997    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.923252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.923235    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.923353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.923339    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.923450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.923438    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.923719 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.923697    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.928552 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.924949    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.935500 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.935390    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.935699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.935675    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.981559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.981469    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxlj\" (UniqueName: \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:42.982035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:42.981984    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:43.014785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:43.014681    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\") pod \"distinct-2-w5-361a44-kserve-7896b6d6bd-q596t\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") " pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:43.087055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:43.087018    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:43.363283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:43.363257    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:11:43.364341 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:43.364318    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a10639a_ee4d_4dbe_a12a_a2573ceedd24.slice/crio-8b4ee373362c85a61722866eb3ec0c9f407de3248840ecb55d9370cf5a740905 WatchSource:0}: Error finding container 8b4ee373362c85a61722866eb3ec0c9f407de3248840ecb55d9370cf5a740905: Status 404 returned error can't find the container with id 8b4ee373362c85a61722866eb3ec0c9f407de3248840ecb55d9370cf5a740905
Nov 02 22:11:44.294892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.294858    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:11:44.299273 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.299245    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.313949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.313924    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:11:44.316511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.316470    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" event={"ID":"1a10639a-ee4d-4dbe-a12a-a2573ceedd24","Type":"ContainerStarted","Data":"5fc70c55992f2995b0477b29da6a35df3eae1269b42b74e09e6f3ba178e4b150"}
Nov 02 22:11:44.316668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.316521    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" event={"ID":"1a10639a-ee4d-4dbe-a12a-a2573ceedd24","Type":"ContainerStarted","Data":"8b4ee373362c85a61722866eb3ec0c9f407de3248840ecb55d9370cf5a740905"}
Nov 02 22:11:44.348446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348405    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.348664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348632    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.348754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348678    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.348842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348823    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.348921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348900    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grm2\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.349000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.348977    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.349045 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.349032    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.349082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.349070    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.349125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.349092    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.352562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.352507    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podStartSLOduration=2.352493831 podStartE2EDuration="2.352493831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:44.351345185 +0000 UTC m=+841.659435991" watchObservedRunningTime="2026-11-02 22:11:44.352493831 +0000 UTC m=+841.660584638"
Nov 02 22:11:44.450776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.450739    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.450986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.450888    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.450986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.450918    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.450986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.450972    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451010    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grm2\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451081    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451121    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451164    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451199    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451326    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451364    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451678    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.451996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451922    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.452186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.451999    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.453419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.453397    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.453697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.453677    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.459602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.459580    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.459920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.459890    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grm2\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.548752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.540799    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:11:44.564556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.563025    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:11:44.564556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.563166    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.617607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.617568    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656208    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656326    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656378    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656422    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656491    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656579    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656604    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656762    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.656788    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tpf\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758472    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758703    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758738    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758892    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758926    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tpf\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.758977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.758957    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.759085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.759033    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.759141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.759082    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.759141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.759123    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.759378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.759354    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.760798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.760770    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.762124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.762098    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.762285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.762119    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.762381 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.762354    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.762647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.762608    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.763689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.763663    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.768370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.768343    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.769147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.769123    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tpf\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.881052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.881005    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:44.954040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.954012    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:11:44.955028 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:11:44.954992    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c80cb_2ad9_4d92_8e7e_256cd29cab28.slice/crio-3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d WatchSource:0}: Error finding container 3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d: Status 404 returned error can't find the container with id 3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d
Nov 02 22:11:44.958133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.958096    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:44.958223 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.958178    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:44.958264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:44.958222    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:45.087692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.087660    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:45.253627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.246001    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:11:45.288363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.268302    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:45.288363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.268396    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:45.288363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.268439    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:11:45.325831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.325793    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" event={"ID":"f89c80cb-2ad9-4d92-8e7e-256cd29cab28","Type":"ContainerStarted","Data":"450f7e31105a3c9c69762f3f9520933f24c027ecf0519e987fc71ac8c11e4803"}
Nov 02 22:11:45.326267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.325863    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" event={"ID":"f89c80cb-2ad9-4d92-8e7e-256cd29cab28","Type":"ContainerStarted","Data":"3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d"}
Nov 02 22:11:45.328090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.328054    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" event={"ID":"fe0c61aa-02f5-42f0-84bd-76fb386690b9","Type":"ContainerStarted","Data":"3e3ca45cd4da578d0d1afea0fd04930683a0ca6e756c53c545f11d57f55bb296"}
Nov 02 22:11:45.346023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.345975    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podStartSLOduration=1.3459610739999999 podStartE2EDuration="1.345961074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:45.343165099 +0000 UTC m=+842.651255905" watchObservedRunningTime="2026-11-02 22:11:45.345961074 +0000 UTC m=+842.654051878"
Nov 02 22:11:45.618193 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:45.618155    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:46.341874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.341838    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" event={"ID":"fe0c61aa-02f5-42f0-84bd-76fb386690b9","Type":"ContainerStarted","Data":"2ccb8c8f7f684a4908accfee337a3e36a3cf92caeb6bbc9c14641342bcbf1c71"}
Nov 02 22:11:46.387556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.387415    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podStartSLOduration=2.387392797 podStartE2EDuration="2.387392797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:46.377111968 +0000 UTC m=+843.685202772" watchObservedRunningTime="2026-11-02 22:11:46.387392797 +0000 UTC m=+843.695483607"
Nov 02 22:11:46.628559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.620394    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:46.628559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.620477    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:46.885619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.885085    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:46.885619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.885231    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:46.885619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.885278    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:46.997558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:46.988770    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:11:47.024185 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.023795    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:11:47.024348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.024118    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.028645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.028612    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"distinct-2-w0-310bb3-kserve-self-signed-certs\""
Nov 02 22:11:47.239656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239453    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.239656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239601    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.239656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239637    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.239991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239708    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fgts\" (UniqueName: \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.239991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239879    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.240119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.239993    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.342562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.342184    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.343076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.342797    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.343076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.342911    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.343076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.342964    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.346610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.345419    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.362571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.351592    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fgts\" (UniqueName: \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.362571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.351794    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.362571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.351981    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.362571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.352550    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.362571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.360242    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.381556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.381242    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fgts\" (UniqueName: \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.407588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.407287    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\") pod \"distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.407765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.407590    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:47.618407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.618363    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:47.618570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.618438    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:47.790631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.790606    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:11:47.882432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.881960    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:47.882432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:47.882021    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:48.420033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.419996    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" event={"ID":"e691aebd-6ff9-4836-8e7c-0f74f7bef077","Type":"ContainerStarted","Data":"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"}
Nov 02 22:11:48.420486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.420043    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" event={"ID":"e691aebd-6ff9-4836-8e7c-0f74f7bef077","Type":"ContainerStarted","Data":"8d5dff0d9185a44688e99a288b4abf8c6617caed52b7d94e54d9cbbd136c550a"}
Nov 02 22:11:48.483927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.481827    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podStartSLOduration=2.460519661 podStartE2EDuration="2.460519661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:48.456869304 +0000 UTC m=+845.764960111" watchObservedRunningTime="2026-11-02 22:11:48.460519661 +0000 UTC m=+845.768610468"
Nov 02 22:11:48.622126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.622075    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:48.622411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.622366    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:48.883579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.882847    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:48.883579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:48.882912    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:49.408033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:49.407993    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:49.618243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:49.618204    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:49.618670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:49.618264    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:49.883558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:49.882980    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:49.883558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:49.883044    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:50.617915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:50.617878    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:50.618188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:50.618163    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:50.883489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:50.882972    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:50.883489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:50.883038    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:51.618619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:51.618581    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 22:11:51.618805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:51.618644    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 22:11:51.881653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:51.881555    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 22:11:51.881902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:51.881626    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 22:11:52.689198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.689163    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:52.690305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.690278    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:52.692892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.692873    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:11:52.892707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.892675    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:52.893029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.893008    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:52.894118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:52.894090    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:11:53.102175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:53.102137    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:53.231579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:53.231525    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:53.670563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:53.670030    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:11:57.408780 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:57.408691    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:57.421394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:57.421369    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:57.496265 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:11:57.496232    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:11:57.562346 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:57.562311    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:11:59.581956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.581913    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:11:59.586211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.586185    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.597632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.597602    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:11:59.633377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633338    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjp2\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633411    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633547    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633599    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633706    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633794 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633729    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633805    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633862    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.633936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.633880    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.651389 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.651348    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:11:59.656887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.656862    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:11:59.657019 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.656998    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.661752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.661728    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.675306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.675280    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:11:59.681095 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.679522    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:11:59.735725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735692    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.735907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735744    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.735907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735780    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.735907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735802    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.735907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735819    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.735907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735877    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735913    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkmc\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735976    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.735999    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736035    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjp2\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736076    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736100    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736167    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736185    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736219    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736287    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736347    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736369    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736384    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736403    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736458    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736560    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59ww\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736594    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736637    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736681    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736732    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.736913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736802    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.736913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736853    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.736913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736882    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.737024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.736974    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.737118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.737099    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.737192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.737176    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.738164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.738143    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.738908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.738892    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.745668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.745628    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjp2\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.745857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.745836    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.838834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838743    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59ww\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.838834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838788    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.838834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838826    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838869    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838891    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.838982    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839058    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839128    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839169    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839194    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkmc\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839258    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839296    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839294    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839338    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839409    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839472    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839507    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839588    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839603    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839688    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.839813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839715    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.840209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839884    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.840209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839961    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.840209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.839993    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.840387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.840253    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.840387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.840335    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.840502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.840431    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.840611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.840586    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.841730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.841707    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.842454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.842427    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.842868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.842848    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.843068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.843050    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.850541 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.850496    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59ww\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.852380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.852355    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkmc\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.853400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.853380    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:11:59.853483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.853451    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.910068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.910019    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:11:59.993712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.993418    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:11:59.996993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:11:59.996701    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:12:00.223809 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:00.223769    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a215371_3f8c_43f5_b22d_ab9d0c6895c3.slice/crio-dd1a5cb1376e5ae737668bc2b2c7b3714f27b418f278ca673e8f7f50c72bf14c WatchSource:0}: Error finding container dd1a5cb1376e5ae737668bc2b2c7b3714f27b418f278ca673e8f7f50c72bf14c: Status 404 returned error can't find the container with id dd1a5cb1376e5ae737668bc2b2c7b3714f27b418f278ca673e8f7f50c72bf14c
Nov 02 22:12:00.223961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.223909    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:12:00.226467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.226433    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.226819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.226778    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.226973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.226836    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.416697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.406062    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:12:00.457855 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:00.457226    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99242638_804b_46e5_ac4e_cd57dfef944f.slice/crio-a79f61614167130ec7e58a86ac462c9e4a53e32bc569b1771f0cf2c6409cc61c WatchSource:0}: Error finding container a79f61614167130ec7e58a86ac462c9e4a53e32bc569b1771f0cf2c6409cc61c: Status 404 returned error can't find the container with id a79f61614167130ec7e58a86ac462c9e4a53e32bc569b1771f0cf2c6409cc61c
Nov 02 22:12:00.465407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.460274    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.465407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.460353    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.465407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.460390    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.465407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.464236    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:12:00.478648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.478613    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.478762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.478691    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.478762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.478724    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:00.623561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.623159    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" event={"ID":"7a215371-3f8c-43f5-b22d-ab9d0c6895c3","Type":"ContainerStarted","Data":"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"}
Nov 02 22:12:00.623561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.623225    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" event={"ID":"7a215371-3f8c-43f5-b22d-ab9d0c6895c3","Type":"ContainerStarted","Data":"dd1a5cb1376e5ae737668bc2b2c7b3714f27b418f278ca673e8f7f50c72bf14c"}
Nov 02 22:12:00.631553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.628907    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" event={"ID":"99242638-804b-46e5-ac4e-cd57dfef944f","Type":"ContainerStarted","Data":"a37407eb222607fa1987168e16e933ef6138edf389515e4f5b145e55bd24db8d"}
Nov 02 22:12:00.631553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.629001    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" event={"ID":"99242638-804b-46e5-ac4e-cd57dfef944f","Type":"ContainerStarted","Data":"a79f61614167130ec7e58a86ac462c9e4a53e32bc569b1771f0cf2c6409cc61c"}
Nov 02 22:12:00.632852 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.632120    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" event={"ID":"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e","Type":"ContainerStarted","Data":"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"}
Nov 02 22:12:00.632852 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.632168    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" event={"ID":"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e","Type":"ContainerStarted","Data":"20a44b5ac5694cead6751c39a3445b3bcbf6ffde963a83d524bb5d54ee53cb8d"}
Nov 02 22:12:00.645718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.645664    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podStartSLOduration=1.645645788 podStartE2EDuration="1.645645788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.643887734 +0000 UTC m=+857.951978540" watchObservedRunningTime="2026-11-02 22:12:00.645645788 +0000 UTC m=+857.953736588"
Nov 02 22:12:00.669618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.668529    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podStartSLOduration=1.668509692 podStartE2EDuration="1.668509692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.666799888 +0000 UTC m=+857.974890691" watchObservedRunningTime="2026-11-02 22:12:00.668509692 +0000 UTC m=+857.976600498"
Nov 02 22:12:00.685438 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.684991    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podStartSLOduration=1.6849738539999999 podStartE2EDuration="1.684973854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.684070357 +0000 UTC m=+857.992161161" watchObservedRunningTime="2026-11-02 22:12:00.684973854 +0000 UTC m=+857.993064664"
Nov 02 22:12:00.910850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.910813    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:12:00.993583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.993492    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:12:00.997683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:00.997653    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:12:01.916572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:01.916511    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:01.917093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:01.916914    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:02.003010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.002666    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:02.003010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.002731    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:02.012635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.011705    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:02.012635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.011807    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:02.930558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.912817    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:02.930558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.912882    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:02.997525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.997485    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:02.998660 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.998631    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:02.999161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.998559    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:02.999328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:02.999306    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:03.915192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:03.915153    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:03.915470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:03.915441    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:03.998594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:03.995702    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:03.998594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:03.995764    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:04.009563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.002706    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:04.009563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.002763    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:04.911135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.911096    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:04.911339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.911162    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:04.998557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.998066    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:04.998557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.998137    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:04.999152 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.998878    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:04.999152 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:04.998936    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:05.911208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:05.911061    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:05.911208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:05.911162    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:06.011935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.010352    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:06.011935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.010410    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:06.020718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.020685    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:06.020980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.020952    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:06.912150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.912106    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:06.912489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.912454    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:06.999562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.998479    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:06.999562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.998575    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:06.999562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.999420    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:06.999562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:06.999461    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:07.911186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.910699    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 22:12:07.911186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.910780    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 22:12:07.994368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.994326    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 22:12:07.994573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.994402    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 22:12:07.997771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.997736    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 22:12:07.997932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:07.997816    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 22:12:09.001506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.001470    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:12:09.005726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.005276    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:12:09.006780 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.006138    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:12:09.011252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.011227    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:12:09.011743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.011722    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:12:09.013365 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.013341    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:12:09.212934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.212900    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:12:09.213557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.213515    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:12:09.214782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:09.214754    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:12:14.437283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.437242    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:12:14.443583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.443557    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:12:14.443737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.443720    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.448064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.448030    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.462523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.462483    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:12:14.465583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.465561    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:12:14.528469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.528437    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:12:14.532919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.532892    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:12:14.533054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.533001    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.537889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.537865    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.542925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.542896    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:12:14.550876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.550851    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:12:14.590341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590306    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.590341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590344    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590459    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.590628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590553    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrj5g\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590601    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590692    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590730    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.590788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590745    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590811    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.590908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590830    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.590908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590848    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.590908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590900    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.591074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590921    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.591074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.590962    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.591173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.591072    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.591173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.591117    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.591238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.591173    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztf6\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.591238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.591213    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.692306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692199    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692268    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692306    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5whr\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692334    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692364    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692408    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692436    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.692513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692477    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692528    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692600    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692630    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692661    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztf6\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692721    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692753    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692811    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692821    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692836    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692865    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.692924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.692905    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.693696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.693665    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.693850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.693771    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.693936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.693873    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.693936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.693913    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.694065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694020    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694034    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694072    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694105    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.694174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694133    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694159    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.694402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694207    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694418    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694460    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.694517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694481    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.694517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694509    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlg2q\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.694762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694573    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694613    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694737    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.694933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694790    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694819    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694879    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrj5g\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.694933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694891    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.694933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694906    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.695182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694945    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.695182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.694950    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.695182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.695143    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.695428 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.695402    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.695787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.695758    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.696741 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.696718    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.697051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.697029    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.697130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.697034    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.703303 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.703276    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.703411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.703384    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztf6\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.703929 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.703907    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrj5g\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.704304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.704283    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.771411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.771375    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:12:14.776865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.776839    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.784472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.784209    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:14.786769 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.786714    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:12:14.787316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.787288    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:14.796417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796392    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796445    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796471    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796504    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5whr\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796524    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796568    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796593    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796627    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796678    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796721    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796740    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796770    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796803    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796832    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796858    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796918    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796948    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.796997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.796990    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlg2q\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.798451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.798424    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.799300    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.799598    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.799871    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.800010    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.800113    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.800346    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.800619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.800617    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.801231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.801210    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.801870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.801812    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.802197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.802171    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.802521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.802499    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.803950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.803929    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.805405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.804782    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.805405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.805356    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlg2q\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.807865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.807819    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5whr\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.813990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.813943    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.818662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.818634    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-f78m5\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.860559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.860467    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:14.880300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.879186    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:14.898607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898205    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898387    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898452    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jfx\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898513    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898568    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898611    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898666    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898743    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:14.898930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:14.898788    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.009446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.009388    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057318    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057409    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057660    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057739    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jfx\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057824    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057885    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.057958    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.063295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.058024    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.067689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.066833    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.067689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.013619    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.067689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.067524    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.075733 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.068307    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.075733 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.068935    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.075733 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.071710    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.079505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.076289    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.082856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.082826    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jfx\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.083191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.083169    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.116135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.115851    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:15.134120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.132178    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:12:15.137260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.136458    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:12:15.146448 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:15.146387    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ff629e_baab_46c7_a7d5_d04a3f0e414d.slice/crio-eb46870b4e30a3240d2f2bf9b92c4bc647152a7887e14929c8a103c14c2e0af0 WatchSource:0}: Error finding container eb46870b4e30a3240d2f2bf9b92c4bc647152a7887e14929c8a103c14c2e0af0: Status 404 returned error can't find the container with id eb46870b4e30a3240d2f2bf9b92c4bc647152a7887e14929c8a103c14c2e0af0
Nov 02 22:12:15.149272 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:15.148293    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1206c0ba_96a6_44b1_b933_df3f858bed5a.slice/crio-56fe46bbf5a6b06003354a8bc2e7e4256bf62effa37b73660c4b963e4e2cab81 WatchSource:0}: Error finding container 56fe46bbf5a6b06003354a8bc2e7e4256bf62effa37b73660c4b963e4e2cab81: Status 404 returned error can't find the container with id 56fe46bbf5a6b06003354a8bc2e7e4256bf62effa37b73660c4b963e4e2cab81
Nov 02 22:12:15.152832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.152071    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.152832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.152161    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.152832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.152206    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.154578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.154088    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.154578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.154156    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.154578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.154185    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.266525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.266490    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:12:15.272463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.272419    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.272595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.272515    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.272640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.272617    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.349142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.349095    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:12:15.361828 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:15.361787    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b95825_ce66_4f77_8eae_03c35fdd0e4d.slice/crio-2f28407d33ecf030020da2b57708c8f2716e7db251b6dca093cdf44ec313c145 WatchSource:0}: Error finding container 2f28407d33ecf030020da2b57708c8f2716e7db251b6dca093cdf44ec313c145: Status 404 returned error can't find the container with id 2f28407d33ecf030020da2b57708c8f2716e7db251b6dca093cdf44ec313c145
Nov 02 22:12:15.365916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.365344    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.365916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.365437    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.365916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.365485    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.452051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.450885    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:12:15.453058 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:15.452689    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbc4305_7123_41ad_aea4_b34b96d69fec.slice/crio-3d62c1e81f025be73cb7dc08dfaf434e21ebef78878dc75f218ad2c8c13dc119 WatchSource:0}: Error finding container 3d62c1e81f025be73cb7dc08dfaf434e21ebef78878dc75f218ad2c8c13dc119: Status 404 returned error can't find the container with id 3d62c1e81f025be73cb7dc08dfaf434e21ebef78878dc75f218ad2c8c13dc119
Nov 02 22:12:15.456235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.455731    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.456235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.455817    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:15.456235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:15.455871    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:16.132335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.132290    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" event={"ID":"8cbc4305-7123-41ad-aea4-b34b96d69fec","Type":"ContainerStarted","Data":"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"}
Nov 02 22:12:16.132520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.132358    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" event={"ID":"8cbc4305-7123-41ad-aea4-b34b96d69fec","Type":"ContainerStarted","Data":"3d62c1e81f025be73cb7dc08dfaf434e21ebef78878dc75f218ad2c8c13dc119"}
Nov 02 22:12:16.134317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.134287    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" event={"ID":"19b95825-ce66-4f77-8eae-03c35fdd0e4d","Type":"ContainerStarted","Data":"49a9528177939c03ffdce223654aa35eb72c6681be4483126ee5b5ed7aaef752"}
Nov 02 22:12:16.134446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.134337    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" event={"ID":"19b95825-ce66-4f77-8eae-03c35fdd0e4d","Type":"ContainerStarted","Data":"2f28407d33ecf030020da2b57708c8f2716e7db251b6dca093cdf44ec313c145"}
Nov 02 22:12:16.142786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.142695    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" event={"ID":"0ea2f692-29d4-4cff-b852-1e3199d41c6f","Type":"ContainerStarted","Data":"7b6393c6eb2fcfc566a5755b1d7d945305c32be83170bac0e9167204491435f4"}
Nov 02 22:12:16.142786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.142749    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" event={"ID":"0ea2f692-29d4-4cff-b852-1e3199d41c6f","Type":"ContainerStarted","Data":"1bb2a1d03818b90c4bdc4268aded7b631c036642174e4a0b50d80faa679171e8"}
Nov 02 22:12:16.144908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.144882    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" event={"ID":"1206c0ba-96a6-44b1-b933-df3f858bed5a","Type":"ContainerStarted","Data":"788cab08fe2e6575f191cbd75a1248548876a8622ed8882b4622eae8e6c735ce"}
Nov 02 22:12:16.145072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.145060    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" event={"ID":"1206c0ba-96a6-44b1-b933-df3f858bed5a","Type":"ContainerStarted","Data":"56fe46bbf5a6b06003354a8bc2e7e4256bf62effa37b73660c4b963e4e2cab81"}
Nov 02 22:12:16.150767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.150081    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" event={"ID":"b1ff629e-baab-46c7-a7d5-d04a3f0e414d","Type":"ContainerStarted","Data":"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"}
Nov 02 22:12:16.150767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.150129    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" event={"ID":"b1ff629e-baab-46c7-a7d5-d04a3f0e414d","Type":"ContainerStarted","Data":"eb46870b4e30a3240d2f2bf9b92c4bc647152a7887e14929c8a103c14c2e0af0"}
Nov 02 22:12:16.158096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.158049    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podStartSLOduration=2.158032385 podStartE2EDuration="2.158032385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:16.155944825 +0000 UTC m=+873.464035632" watchObservedRunningTime="2026-11-02 22:12:16.158032385 +0000 UTC m=+873.466123183"
Nov 02 22:12:16.208911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.208839    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podStartSLOduration=2.208814514 podStartE2EDuration="2.208814514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:16.1960848 +0000 UTC m=+873.504175606" watchObservedRunningTime="2026-11-02 22:12:16.208814514 +0000 UTC m=+873.516905325"
Nov 02 22:12:16.234458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.234389    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podStartSLOduration=2.234371744 podStartE2EDuration="2.234371744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:16.23370596 +0000 UTC m=+873.541796766" watchObservedRunningTime="2026-11-02 22:12:16.234371744 +0000 UTC m=+873.542462553"
Nov 02 22:12:16.303616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.303549    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podStartSLOduration=2.303515826 podStartE2EDuration="2.303515826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:16.259338124 +0000 UTC m=+873.567428942" watchObservedRunningTime="2026-11-02 22:12:16.303515826 +0000 UTC m=+873.611606633"
Nov 02 22:12:16.311460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.311396    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podStartSLOduration=2.311375229 podStartE2EDuration="2.311375229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:16.298431287 +0000 UTC m=+873.606522094" watchObservedRunningTime="2026-11-02 22:12:16.311375229 +0000 UTC m=+873.619466035"
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.789888    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.790054    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.790096    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.790546    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.791922    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:16.794008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.791963    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:16.874557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.874398    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:16.896557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.881154    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:16.896557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.881219    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:16.896557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.882618    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:16.896557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.887058    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:16.896557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:16.887107    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:17.123561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.120378    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:17.123561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.120513    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:17.123561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.120580    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:17.509757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.509638    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:17.557815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.557739    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:17.790098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.790003    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:17.791842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.791717    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:17.792040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.792012    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:17.792201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.791804    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:17.841997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.841940    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:17.863010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.862973    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:17.863270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.863242    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:17.883000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.882963    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:17.883262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.883237    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:17.966557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.964630    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:17.966557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:17.965580    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:18.017561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.015518    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:18.017561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.016429    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.097423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.095626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:18.119780 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.118750    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.120036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.120013    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:18.120165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.120142    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:18.186684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.186625    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.226180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.226135    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.419689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.419636    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.522687 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.522632    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.523845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.523818    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:12:18.675614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.652755    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:18.732559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.725075    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:12:18.813568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.796154    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:18.813568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.796210    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:18.850559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.822336    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:18.850559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.822394    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:18.851821 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.851235    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.855268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.855234    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:18.915561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.895880    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:18.915561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.895941    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:18.915561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.896013    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:18.915561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.896035    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:18.986820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:18.986715    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:19.105562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.016627    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.105562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.017038    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:19.105562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.055649    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.216566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.199882    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:19.216858 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.216817    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:19.259755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.224048    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.259755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.225963    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.387264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.387216    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.526756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.526645    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655332    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655403    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655440    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655476    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655497    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655575    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655613    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:12:19.656559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.655648    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:12:19.685559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.676884    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:19.713228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.692156    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:19.915608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.874682    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.982642    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.982708    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.983253    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.983279    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.983343    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.001633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:19.983363    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:20.005206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.003566    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.005206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.003627    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:20.038602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.028356    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:20.038602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.028717    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.299274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.299167    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.320558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.300044    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:20.452202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.452145    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:20.473259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.473207    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.503977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.503925    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.529426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.529394    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:12:20.530625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.530588    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.694001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.693950    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.800906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.800852    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:20.801448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.801411    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:20.822396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.822349    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:20.834557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.823280    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.834861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.834828    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:20.835027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.823349    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:20.835139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:20.835116    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:21.059617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.058858    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.086816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.086749    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:21.087080 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.087049    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:21.092624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.090067    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.092624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.090417    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.168607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.161308    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:21.168607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.161364    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:21.168607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.165196    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:21.194559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.165268    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:21.314608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.299928    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:21.314608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.313182    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:21.314608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.313266    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:21.325613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.321634    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:21.517560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.492161    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.618736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.618698    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.620182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.620155    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 02 22:12:21.620645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.620624    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.621060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.621039    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:21.764960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.757818    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:12:21.775350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.775317    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:12:21.830813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.830478    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:21.852322 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.739772    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:21.852322 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.845060    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:21.944615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.922630    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.952124    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.952182    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.965800    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.965856    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.965958    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.965984    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.966920    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.041686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:21.966961    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:22.108770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.061300    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:22.109736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.101616    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:22.110142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.101680    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.292618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.292481    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.292618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.292557    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:22.331087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.329746    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.331087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.330456    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:22.434572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.432175    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:22.764566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.716861    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.764566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.717264    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.809559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.781948    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.838559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.809856    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.838559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.836630    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.838559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.838238    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.838559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.838289    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.838944    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.838981    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.890644    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.898957    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.899014    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.899087    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:22.913575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.899108    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:22.945083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:22.945035    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.119146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.115975    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.119146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.118199    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:23.119146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.118842    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.159726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.159668    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.183766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.183722    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:23.184064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.184018    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:23.339557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.338691    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.343558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.341997    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.720706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.720662    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:23.721314 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.721283    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:23.813755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.813703    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.842122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.814838    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:23.842445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.842417    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:23.842642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.841663    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:23.842834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.814875    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.854559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.815135    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:23.854881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.854836    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:23.946005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.945952    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:23.957303 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.947267    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:23.957655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.957619    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:23.957807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.955589    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:23.957924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:23.957904    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:24.066655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.066552    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:24.098992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.098948    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 02 22:12:24.125162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.119802    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:24.125162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.120291    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:24.125162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.120617    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:24.125162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.121908    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:24.125162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.121943    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:24.422413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.422362    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:24.423557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.423510    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:24.625511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.625459    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:24.995338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.828332    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.015939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.856022    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.017430 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.856402    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.018646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.878748    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.018831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.018802    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:25.018962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.984856    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.020052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.958839    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:25.021142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.898051    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.155750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.147589    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:25.155750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.091681    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:25.195858 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.195686    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:25.204604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.204552    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:25.269575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.994919    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.271310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:24.928611    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.271583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.271520    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:25.271787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.064529    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.271933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.271912    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.081652    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.460084    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.458724    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.458677    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.289749    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.461609    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.391821    4929 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.461676    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.467175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.463192    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.467974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.467910    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.520218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.520122    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:12:25.521415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.521381    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.870484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.855028    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:25.940570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.938071    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.940570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.938138    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:25.940570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.940270    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.940570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.940318    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:25.940940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.940875    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.940940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.940910    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:25.974143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.974107    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:25.974443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:25.974400    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:26.004142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.004100    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.052573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.046491    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.055572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.053115    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.055572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.053476    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.055572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.054617    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.121443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.118714    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:26.121443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.118775    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:26.147857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.147816    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.197051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.196634    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.259559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.246178    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.285557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.269168    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.285557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.271142    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.461932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.460661    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.466876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.465611    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:26.471954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.470699    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.786385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.786289    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:26.786731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.786692    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:26.792512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.790961    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:26.792512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.791006    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:26.863978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.863938    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:26.864358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.864332    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:26.864926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.864240    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:26.890554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.882088    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:26.890554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:26.882169    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:27.007591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.005701    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.061620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.060650    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.061620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.061143    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:27.061620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.061434    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.094963 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.093925    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.118958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.118920    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:27.119230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.119202    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:27.149748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.149682    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.200803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.200739    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.272906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.272823    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.273512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.273479    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:27.461716 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.461657    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.471034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.470927    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.471415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.471269    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:27.785135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.785020    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:27.785427 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.785375    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:27.790579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.790548    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:27.790850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.790813    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:27.819825 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:12:27.816914    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:12:27.861236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.861186    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:27.861583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.861511    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:27.883092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.883058    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:27.883400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:27.883356    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:28.008699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.008649    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.061520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.061403    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.062196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.062147    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.063726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.063147    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.101462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.101407    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:28.117003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.116968    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:28.117314 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.117271    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:28.165858 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.165780    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.202766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.202694    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:28.273759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.273691    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.275283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.275149    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.423726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.423679    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Nov 02 22:12:28.465835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.465791    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.473885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.473842    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:28.475246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.475217    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.781339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.776642    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:28.786178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.786124    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:28.786495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.786447    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:28.796514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.796481    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:28.796861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.796817    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:28.862161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.862125    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:28.862452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.862412    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:28.884013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.883977    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:28.884295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:28.884255    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:29.009702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.009655    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:29.063294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.063189    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.063886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.063856    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.064583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.063956    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.102418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.102336    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.121837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.118633    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:29.121837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.118696    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:29.166672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.166626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.205674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.205618    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:29.230887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.230853    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:12:29.286382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.286333    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.288022 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.287972    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:29.478115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.478068    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:29.478817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.478784    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:29.537565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.504635    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:29.836664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.799063    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:29.994568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.992690    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:29.994568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.805849    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:29.994568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.993215    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:29.994568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.816328    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:29.994568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.993838    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:29.994984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.981604    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:29.994984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:29.994646    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:30.039699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.039651    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:30.065043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.042447    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:30.066049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.066029    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:30.130649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.065312    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.065388    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.065882    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.042752    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.177970    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.169676    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.179709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.131228    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:30.215557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.214687    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.259250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.259210    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.259563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.259511    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:30.260147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.260125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:12:30.308187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.308133    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.310192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.310160    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.408083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.407996    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:12:30.480149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.479822    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:30.630355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.630242    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:30.819634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.817773    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.819634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.817853    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:30.919788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.919752    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.933589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.923663    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 22:12:30.933589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.924461    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.933589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.924506    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 22:12:30.933589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.929986    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:12:30.933927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.931776    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:30.933927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.933656    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:30.956291 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:30.956243    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:31.119815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.119711    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:31.120050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.120025    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:31.186580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.176837    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:12:31.186580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.178057    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:31.186580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.178149    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:31.186580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.178306    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:31.270486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.228616    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:31.357340 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.332194    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:31.361163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.332232    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:31.364818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.364791    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:12:31.384620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.384525    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:31.543606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.504680    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:12:31.543606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.509842    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:31.813917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.809721    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 22:12:31.813917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.809782    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 22:12:31.906769 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.906717    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:12:31.939554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.932028    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 22:12:31.939554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:31.932111    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 22:12:32.003339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.003307    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:12:32.062033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.059792    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:12:32.112690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.108951    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:12:32.136558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.121044    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 22:12:32.136905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.136869    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 22:12:32.544147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.544057    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:32.577568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.562163    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:12:32.770577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.763870    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:32.885569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.883918    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:12:32.885569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.883983    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:32.886524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:32.886489    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:33.003420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.003370    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:33.010234 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.010178    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:12:33.200883 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.196968    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:33.299857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.287227    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:12:33.417350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.417312    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:33.418980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:33.418956    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:12:34.163914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:34.163640    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:12:34.163914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:34.163707    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:34.481358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:34.481277    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:34.483803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:34.483774    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:12:39.588759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:39.588635    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:39.727692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:39.727641    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:39.816695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:39.816642    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:39.878688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:39.878626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.036249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.035321    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.036249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.035990    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.075686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.075631    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.098565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.097625    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.298946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.298842    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.331680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.331621    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.333125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.333093    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.496680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.496631    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.589840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.589730    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.590663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.590636    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:12:40.768348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.743642    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.768348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.744431    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:12:40.786957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.770186    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:40.944977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.914002    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:40.946277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:40.923472    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.043181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.043135    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.069556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.044834    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.098776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.098722    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:41.249624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.078720    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:12:41.249938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.249916    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:12:41.250117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.250102    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:12:41.250269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.250255    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:12:41.266200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.250394    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:12:41.267209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.076818    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:41.267629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.234982    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.267850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.267831    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:12:41.304581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.299669    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.349656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.309128    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:41.349972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.349936    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:41.379923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.379890    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:12:41.381191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.381156    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.381800 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.381779    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:12:41.501069 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.497684    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.501069 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.498340    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:12:41.593208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.593166    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:41.943561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.911710    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.247763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.945824    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.248766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:41.947561    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.249267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.116114    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.278604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.260644    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.279261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.279194    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:42.433558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.429294    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.438685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.438640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:42.469714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.469658    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 02 22:12:42.507621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.471068    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.907558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.568039    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:42.907558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.613196    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:42.907558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:42.678217    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.265616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.249656    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.265616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.249761    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.265616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.249808    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.265616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.261651    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.295557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.281649    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.295557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.281759    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.422667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.422607    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.448718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.448676    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.458318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.448629    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.848561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.848116    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.915562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.899705    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:43.923798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.923746    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:43.924423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:43.924377    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.151056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.134699    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:44.480618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.454740    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.481960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.481920    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:44.491684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.480687    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:44.537559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.492439    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.605557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.492472    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.605557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.604686    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:44.605557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.492516    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.634565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.614697    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:44.634565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.492496    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:44.777225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:44.777139    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:12:45.013566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.012644    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:45.071610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.070423    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:12:45.090016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.089964    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:45.265884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.262432    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:12:45.265884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.262499    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:12:45.265884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.262598    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:12:45.265884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.262618    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:12:45.275862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.275817    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:12:45.276837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.276819    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.278611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.278588    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:12:45.293792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.293768    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:12:45.319045 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.319007    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:12:45.320068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.320041    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:45.325848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.325749    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351465    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351686    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351754    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351801    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351823    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:12:45.352320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.351865    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:12:45.356007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.354822    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.363753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.363728    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:12:45.365406 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.363831    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.394589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394549    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.394772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394657    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.394772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394687    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.394772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394745    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.394951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394858    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.394951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.394928    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.395070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.395027    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgp4\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.395806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.395170    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.397240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.397216    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.405730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.405626    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:12:45.412592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.409201    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:12:45.498330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498295    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.498330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498335    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.498522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498412    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.498522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498451    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.498643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498599    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.498679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498668    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.498713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498687    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg2w\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.498920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498904    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.498983    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499003    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499166 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499101    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.499231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499160    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499190    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.499231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499214    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.499445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499240    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.499445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499293    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499294    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499750    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.499839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499817    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.499889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.499861    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgp4\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.500117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500087    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.500245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500126    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.500245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500156    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.500245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500168    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.500245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500227    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500267    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500295    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500340    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500368    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500385    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.500470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500435    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.500987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.500966    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.503100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.502801    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.503594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.503571    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.508588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.508548    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgp4\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.509246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.509223    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.602605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602570    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602610    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602645    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602690    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602714    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602747    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.602789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602768    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602856    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602881    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg2w\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.602986    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603017    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603041    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603070    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603081    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603152    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603203    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603341    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603362    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603373    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603410    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603415    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.603508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.603464    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.604419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.604394    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.604528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.604483    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.604648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.604592    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.604760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.604736    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.605000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.604970    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.605236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.605206    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.607939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.607526    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.607939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.607635    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.609062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.609034    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.612111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.612083    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.612209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.612161    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg2w\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.612862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.612828    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.613363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.613341    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.614831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.614807    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:45.657748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.657720    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:45.679681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.679649    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:45.705548 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:45.690308    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:46.049167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.049121    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:12:46.109908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.101067    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.109908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.101163    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.109908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.101214    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.109908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.104857    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:12:46.109908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.108634    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:12:46.111581 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:12:46.111511    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc6f219_bc05_4252_87d1_535f3dffee4c.slice/crio-fcb2601840c673a1b73ed254583a0bc0081139ec57d2aaa2d3ad85ded318249e WatchSource:0}: Error finding container fcb2601840c673a1b73ed254583a0bc0081139ec57d2aaa2d3ad85ded318249e: Status 404 returned error can't find the container with id fcb2601840c673a1b73ed254583a0bc0081139ec57d2aaa2d3ad85ded318249e
Nov 02 22:12:46.117005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.116971    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.117110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.116998    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.117110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.117039    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.117110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.117061    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.117110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.117069    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.117110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.117089    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:12:46.360637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.358899    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" event={"ID":"bbc6f219-bc05-4252-87d1-535f3dffee4c","Type":"ContainerStarted","Data":"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"}
Nov 02 22:12:46.360637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.358975    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" event={"ID":"bbc6f219-bc05-4252-87d1-535f3dffee4c","Type":"ContainerStarted","Data":"fcb2601840c673a1b73ed254583a0bc0081139ec57d2aaa2d3ad85ded318249e"}
Nov 02 22:12:46.362574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.361446    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" event={"ID":"6bda60b1-00f8-4dc5-af5e-9459abcfe271","Type":"ContainerStarted","Data":"88b5f3ccb9203c86849451b823d7452f535268153df846770025bfa5ac811031"}
Nov 02 22:12:46.362574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.361491    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" event={"ID":"6bda60b1-00f8-4dc5-af5e-9459abcfe271","Type":"ContainerStarted","Data":"81051875906f7893032527192e030b1801356c8dc2b77b0a267835904dae5088"}
Nov 02 22:12:46.365603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.364852    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" event={"ID":"165850e3-d69d-414d-906a-67cb59e15405","Type":"ContainerStarted","Data":"2afa3e4b077949cf03bc8b98e06ec0c992af7de15847f59e0d50d37ff56ba67d"}
Nov 02 22:12:46.365603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.364903    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" event={"ID":"165850e3-d69d-414d-906a-67cb59e15405","Type":"ContainerStarted","Data":"8038784e3817283f865178a19b854d2c59c08526922c9dd76be7905ccaa26689"}
Nov 02 22:12:46.403136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.403074    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podStartSLOduration=2.403055035 podStartE2EDuration="2.403055035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:46.399852451 +0000 UTC m=+903.707943261" watchObservedRunningTime="2026-11-02 22:12:46.403055035 +0000 UTC m=+903.711145850"
Nov 02 22:12:46.406123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.406071    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podStartSLOduration=2.406060245 podStartE2EDuration="2.406060245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:46.379983951 +0000 UTC m=+903.688074757" watchObservedRunningTime="2026-11-02 22:12:46.406060245 +0000 UTC m=+903.714151051"
Nov 02 22:12:46.423860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.423800    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podStartSLOduration=2.423780739 podStartE2EDuration="2.423780739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:46.423670313 +0000 UTC m=+903.731761131" watchObservedRunningTime="2026-11-02 22:12:46.423780739 +0000 UTC m=+903.731871547"
Nov 02 22:12:46.658136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.658061    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:46.680584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.680557    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:46.690880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:46.690845    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:47.659557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.659448    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:47.659557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.659518    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:47.688558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.683990    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:47.688558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.684053    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:47.708560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.707672    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:47.708560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:47.707749    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:48.658959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.658923    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:48.659269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.659239    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:48.693880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.693843    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:48.694512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.694465    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:48.694687 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.693854    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:48.694834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:48.694814    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:49.662212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.662172    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:49.662608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.662574    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:49.713029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.712981    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:49.715009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.714975    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:49.715555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.713701    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:49.715755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:49.715731    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:50.667575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.664951    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:50.667575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.665019    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:50.699559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.688722    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:50.699559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.688799    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:50.699559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.692888    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:50.699559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:50.692943    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:51.727562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.725440    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:51.727562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.725594    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:51.727562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.726032    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:51.727562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.726070    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:51.729682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.728364    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:51.729682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:51.728418    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:52.667763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.667709    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:52.674592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.668035    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:52.711822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.711779    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:52.715690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.715643    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:52.715889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.712121    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:52.716326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:52.716073    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:53.658728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.658687    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 22:12:53.659205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.658750    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 22:12:53.681468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.681423    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 22:12:53.681695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.681485    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 22:12:53.691822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.691785    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 22:12:53.691959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:53.691845    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 22:12:55.191064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.191024    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:55.192590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.192410    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:55.193179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.193156    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:12:55.252461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.252427    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:55.254415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.254391    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:55.274228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.274192    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:12:55.283949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.283915    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:55.285295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.285268    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:55.292459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.292381    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:12:55.873508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:55.873472    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:12:57.851561 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:12:57.841523    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:12:58.259752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.259636    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.290558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.287662    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.317027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.314473    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.389591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.369747    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.421693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.421635    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.452590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.442363    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.477559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.472582    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.488489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.487637    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.602712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.601605    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.612558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.611053    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:12:58.778559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.777631    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.861622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.857634    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.889559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.888670    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:58.903558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:58.893954    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.266623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.265626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.266623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.266452    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:12:59.420558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.420017    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.456581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.430772    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.456581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.431663    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.456581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.432805    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:12:59.456581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.432863    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:12:59.456581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.432897    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:12:59.524620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.522768    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.524620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.523256    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:12:59.524928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.524712    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.559657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.525062    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:12:59.581589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.559968    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.581589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.578733    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:12:59.623557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.614620    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.623557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.615066    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:12:59.734469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.734424    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.735081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.735057    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:12:59.893558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.871243    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:12:59.893558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.872488    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:12:59.907645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.907607    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:12:59.996560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.910423    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.073477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:12:59.997227    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:00.098910 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.098710    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:00.234608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.215894    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:13:00.234608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.225960    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.285556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.279597    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.348816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.348775    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.366559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.363264    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.389112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.366934    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.443353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.443312    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.445761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.445718    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.447083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.447046    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.486807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.486616    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.524696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.524654    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.553573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.548312    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.553573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.548389    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.585289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.556641    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.635560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.632100    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.678681    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.678863    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.678918    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.678992    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.679029    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.679215    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvnz\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.679286    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.679489    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.679559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.679569    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.737683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.717964    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.772653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.717893    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:00.789872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.789835    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.796565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.741386    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.919560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.893169    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.960070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.902824    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.993615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.902861    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 22:13:00.993615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.732528    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.993615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.931976    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:00.993615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.956562    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:00.993615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:00.958416    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.006557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.002674    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvnz\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.006557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.002779    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.041559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.040780    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.070561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.063581    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvnz\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.216215 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.090270    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.228958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.228923    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.508552 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.508482    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.556561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.514000    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:13:01.556561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.542217    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:13:01.556561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.542475    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:01.557660 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.556980    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.644562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.606204    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.645485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.645444    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.720570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.699669    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.794614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.782298    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.794614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.785862    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:13:01.794614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.785898    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:13:01.794614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.786394    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-flv2h\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.821918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.821874    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:01.837365 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.837333    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:13:01.982571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.982029    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:01.998623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:01.987778    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:02.092381 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.091480    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:02.224555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.009936    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:02.273970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.256343    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:13:02.273970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.256521    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:13:02.281753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.278614    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:02.281753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.278704    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:13:02.316524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.316439    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:13:02.328343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.321265    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:13:02.402427 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.401832    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:13:02.416017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.415992    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.440658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.439806    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:13:02.543479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.533731    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 02 22:13:02.543479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.542886    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:02.559239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.559187    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:02.582195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.582118    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:13:02.598031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.590818    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:02.598031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.594020    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:13:02.634704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.634672    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675413    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675464    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675520    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675607    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675657    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675694    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675794    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675853    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.676517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.675934    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wdn\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777474    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777518    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777579    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777686    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777739    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.777896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777774    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777908    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.777978    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.778088    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wdn\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.778272    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.778596    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.778936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.778877    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.779015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.778955    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.779015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.779003    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.780060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.780037    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.780859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.780838    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.785312 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.785288    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.786274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.786251    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wdn\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:02.861098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.861068    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:13:02.862079 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:02.862053    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f131e84_9602_4858_871f_7547019f4e96.slice/crio-ac9db5d1f5c61256cfd2196ee2353693c0d761629ba883f3eea96ae6a2e0c3cf WatchSource:0}: Error finding container ac9db5d1f5c61256cfd2196ee2353693c0d761629ba883f3eea96ae6a2e0c3cf: Status 404 returned error can't find the container with id ac9db5d1f5c61256cfd2196ee2353693c0d761629ba883f3eea96ae6a2e0c3cf
Nov 02 22:13:02.864887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.864855    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:02.864965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.864922    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:02.864965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.864951    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:02.936130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:02.936102    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:03.273488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.269686    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:13:03.299568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.299011    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:03.299568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.299132    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:03.299568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.299195    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:03.581511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.581420    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" event={"ID":"ceb72af2-8e22-42c1-8633-b9ef19ca0b01","Type":"ContainerStarted","Data":"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"}
Nov 02 22:13:03.581511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.581471    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" event={"ID":"ceb72af2-8e22-42c1-8633-b9ef19ca0b01","Type":"ContainerStarted","Data":"f3e4963048f460ab32a1688bb649491408d931c7b46a47040d128c3e1c73ed40"}
Nov 02 22:13:03.584625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.584592    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" event={"ID":"3f131e84-9602-4858-871f-7547019f4e96","Type":"ContainerStarted","Data":"53af4bc8a6502bcc24df705fabc786c04f62826cfa4c389d7a0fc2544829dc22"}
Nov 02 22:13:03.584787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.584644    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" event={"ID":"3f131e84-9602-4858-871f-7547019f4e96","Type":"ContainerStarted","Data":"ac9db5d1f5c61256cfd2196ee2353693c0d761629ba883f3eea96ae6a2e0c3cf"}
Nov 02 22:13:03.602042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.601991    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podStartSLOduration=4.601977792 podStartE2EDuration="4.601977792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:03.599466118 +0000 UTC m=+920.907556924" watchObservedRunningTime="2026-11-02 22:13:03.601977792 +0000 UTC m=+920.910068600"
Nov 02 22:13:03.619545 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.619484    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podStartSLOduration=4.6194706 podStartE2EDuration="4.6194706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:03.61738015 +0000 UTC m=+920.925470958" watchObservedRunningTime="2026-11-02 22:13:03.6194706 +0000 UTC m=+920.927561405"
Nov 02 22:13:03.827891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.827855    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:03.829144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.829110    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:03.829285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.829178    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:03.973462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:03.939670    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:04.827194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:04.827161    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:04.827620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:04.827208    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:04.962592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:04.961659    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:04.962592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:04.961729    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:05.826691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:05.826653    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:05.826870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:05.826715    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:05.957560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:05.950961    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:05.957560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:05.951036    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:06.826403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:06.826367    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:06.826598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:06.826422    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:06.963747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:06.963709    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:06.964238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:06.963782    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:07.827146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:07.827112    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:07.827325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:07.827171    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:07.942057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:07.942015    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:07.942353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:07.942326    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:08.827097 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:08.827061    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:08.827500 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:08.827134    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:08.957560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:08.957017    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:08.957560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:08.957086    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:09.826601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:09.826518    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 22:13:09.826855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:09.826612    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 22:13:09.937297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:09.937252    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:09.937849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:09.937339    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:10.944553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:10.944493    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 22:13:10.945057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:10.944590    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 22:13:11.065196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.065163    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:11.066151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.066125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:11.067749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.067726    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:13:11.941073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.941043    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:11.941336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.941318    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:11.942172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:11.942152    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:13:15.828559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:15.823648    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:15.828559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:15.824145    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:15.865558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:15.855906    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.109559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.107128    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.185558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.184937    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Nov 02 22:13:16.318556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.315629    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.458619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.455263    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.458619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.456080    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.549078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.545648    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.554566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.553361    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:13:16.812620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.809661    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 02 22:13:16.836566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.834607    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.893564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.889819    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:13:16.909561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.894637    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:16.909561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.895281    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:13:16.974208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:16.974164    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.085640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.065361    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.109009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.108956    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.110563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.110521    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:13:17.326559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.316034    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.326559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.316463    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:13:17.491515 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.491073    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.532846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.491971    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:13:17.634108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.524640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.634108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.551526    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.634108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.611310    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:13:17.634108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.611375    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:13:17.640654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.634605    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.640654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.636023    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:13:17.653453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.653413    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:17.679995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.668625    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.905560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.887350    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.909456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.896618    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.974694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.974651    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:17.976334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.976305    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:13:17.989815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.987422    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:17.989815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:17.988562    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.068702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.068652    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.072702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.072678    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:18.137755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.137709    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.317030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.316922    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.523737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.515287    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:13:18.526482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.506717    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.647561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.638184    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.647856    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.664590    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.682685    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.682974    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.683437    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.683494    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.685979    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:18.707596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.686019    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.938612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.933294    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.951421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.951369    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:18.980760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.962282    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.020563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:18.977367    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.082561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.069617    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:13:19.142565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.141744    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:13:19.167359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.167318    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:19.171720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.171680    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.319604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.319483    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:19.319856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.319819    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.354570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.352631    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.528559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.523819    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.584643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.578640    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:19.584643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.578734    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.653555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.647664    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.656598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.656554    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.695516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.665655    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.696919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.686657    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.698501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.686718    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:19.941757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.940162    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:19.961680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:19.958960    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.107558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.102014    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.107558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.102406    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:20.162555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.161950    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.356562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.354626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.568600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.568555    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.683512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.654343    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.701131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.701087    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.702452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.702405    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:20.702911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.702883    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:20.713651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.713607    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.794656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.793085    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:13:20.796919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.796844    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:20.842072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.842041    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:20.842351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.842337    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:13:20.887794 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:20.886674    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:13:21.027923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.027832    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:13:21.070044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.070016    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:21.070295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.070283    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:13:21.070402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.070394    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:13:21.075642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.075619    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:13:21.083110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.083088    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:13:21.083299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.083286    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:13:21.083484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.083474    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:13:21.684263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.684230    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:13:21.692167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.692137    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.714247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.709430    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ec62fcf4-openshift-default-dockercfg-ffkcm\""
Nov 02 22:13:21.714247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.712447    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.731914    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.731957    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732092    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732111    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732157    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732237    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732296    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732484    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.733589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.732576    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834095    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834133    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834321    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834398    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834452    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834628    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834643    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834742    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834816    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834852    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.834891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834869    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.835113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.834992    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.835414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.835382    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.835517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.835501    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.837105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.837082    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.837285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.837270    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.842154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.842126    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:21.843086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:21.843067    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\") pod \"e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") " pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:22.054678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:22.054526    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:22.564029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:22.563898    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:13:22.564912 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:22.564879    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1652513_8ac1_4447_92f2_b8c93430ccbd.slice/crio-f284fa6cebbcfa91a617f9a2a984a289c11944234000909f7e4e00e1ad20d2f7 WatchSource:0}: Error finding container f284fa6cebbcfa91a617f9a2a984a289c11944234000909f7e4e00e1ad20d2f7: Status 404 returned error can't find the container with id f284fa6cebbcfa91a617f9a2a984a289c11944234000909f7e4e00e1ad20d2f7
Nov 02 22:13:22.568024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:22.567987    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:22.568110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:22.568089    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:22.568180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:22.568154    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:23.218901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:23.218864    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" event={"ID":"a1652513-8ac1-4447-92f2-b8c93430ccbd","Type":"ContainerStarted","Data":"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"}
Nov 02 22:13:23.219302 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:23.218932    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" event={"ID":"a1652513-8ac1-4447-92f2-b8c93430ccbd","Type":"ContainerStarted","Data":"f284fa6cebbcfa91a617f9a2a984a289c11944234000909f7e4e00e1ad20d2f7"}
Nov 02 22:13:23.239300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:23.239248    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" podStartSLOduration=2.23923453 podStartE2EDuration="2.23923453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:23.236904649 +0000 UTC m=+940.544995461" watchObservedRunningTime="2026-11-02 22:13:23.23923453 +0000 UTC m=+940.547325335"
Nov 02 22:13:24.055320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:24.055289    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:24.060826 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:24.060799    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:24.228380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:24.228350    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:24.229297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:24.229279    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:13:26.014814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.014780    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:13:26.019466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.019441    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.028897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.028868    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ec62fcf4\""
Nov 02 22:13:26.029122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.028974    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:13:26.029122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.029041    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ec62fcf4-dockercfg-4jd97\""
Nov 02 22:13:26.156440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.156396    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx2w\" (UniqueName: \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.156920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.156892    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.262803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.262758    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx2w\" (UniqueName: \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.267433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.267341    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.276600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.276176    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.344695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.344662    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx2w\" (UniqueName: \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\") pod \"payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.354763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.354730    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:26.781050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:26.781016    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:13:26.781318 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:26.781292    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1feec_0fc3_401e_aff6_ba2d507887bd.slice/crio-6205e6fe3fafbee8135cf20e2342ec04a6e28d4791e01e43a827588429590801 WatchSource:0}: Error finding container 6205e6fe3fafbee8135cf20e2342ec04a6e28d4791e01e43a827588429590801: Status 404 returned error can't find the container with id 6205e6fe3fafbee8135cf20e2342ec04a6e28d4791e01e43a827588429590801
Nov 02 22:13:27.293394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:27.293355    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" event={"ID":"91e1feec-0fc3-401e-aff6-ba2d507887bd","Type":"ContainerStarted","Data":"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"}
Nov 02 22:13:27.293814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:27.293438    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" event={"ID":"91e1feec-0fc3-401e-aff6-ba2d507887bd","Type":"ContainerStarted","Data":"6205e6fe3fafbee8135cf20e2342ec04a6e28d4791e01e43a827588429590801"}
Nov 02 22:13:27.293814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:27.293459    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:27.310691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:27.310638    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" podStartSLOduration=2.310624204 podStartE2EDuration="2.310624204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:27.309323602 +0000 UTC m=+944.617414407" watchObservedRunningTime="2026-11-02 22:13:27.310624204 +0000 UTC m=+944.618715008"
Nov 02 22:13:27.531557 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:13:27.528897    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:13:35.877375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:35.877335    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:13:37.951714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:37.951653    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.118889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.118818    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.134734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.134689    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.352616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.351601    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:13:38.385568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.381305    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.419619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.419481    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.614609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.609510    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.744558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.743099    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:13:38.744558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.744437    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.776943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.746502    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:38.776943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.750951    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:38.776943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.751258    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:38.843036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.842965    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 22:13:38.974649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:38.972715    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.101561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.055025    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.203684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.203629    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.254224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.254095    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.292900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.263286    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:13:39.413567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.382782    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.578626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.578284    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:13:39.578626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.578359    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:13:39.578626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.578399    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:39.578626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.578433    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:13:39.614559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.610763    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.629895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.629860    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:13:39.756714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.756670    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:39.758288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.758254    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:39.759618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.759591    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:39.760752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.760721    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.798698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.761272    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:39.800688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.800662    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:13:39.800954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.800939    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:13:39.801117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.801104    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:13:39.801474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.761315    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:39.811685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.811655    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:13:39.999559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:39.974586    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.234565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.205615    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.235038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.234997    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.295096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.271334    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:13:40.295096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.272410    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:40.295096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.274002    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:13:40.295096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.289095    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.447560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.391168    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:13:40.706572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.706038    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:13:40.770590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.769664    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.770590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.770041    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.770590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.770305    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.826887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.774832    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.826887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.822156    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:40.826887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:40.823698    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.000458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.000357    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:13:41.270631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.235159    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.283406 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.273713    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.306696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.306638    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.382560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.356412    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.501560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.356614    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.529680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.403171    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:41.529985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.529946    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.530426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.490859    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.712702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.712640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.816611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.774010    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:41.920561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.774693    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:41.921847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.774749    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:41.924912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.924871    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:13:41.941558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:41.925104    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.326669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.008827    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.331596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.232433    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:42.331888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.331838    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:42.365313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.365258    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.380122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.365581    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.431161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.409945    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.432697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.432628    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:42.497560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.480698    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:13:42.530916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.530862    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:42.771562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:42.768319    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:43.057617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.051659    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.057617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.052088    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.057617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.052403    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.057617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.052698    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.185560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.163660    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Nov 02 22:13:43.451964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.451917    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.489703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.489654    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:43.646599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.603294    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.866959    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.640225    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.640099    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.640251    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.640393    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.686991    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:13:43.899619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:43.822269    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.083567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.079738    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.122013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.086423    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.150612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.086495    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.180563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.086584    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.198557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.194967    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 02 22:13:44.485615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.482671    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.519700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.506188    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.597981 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:13:44.592337    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.778s"
Nov 02 22:13:44.671556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.671335    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.898073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.872340    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.903973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.874619    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.904553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.874678    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:44.905040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.874705    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.908905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.901294    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:13:44.924741 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:44.901343    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.124560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.122251    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.124560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.123206    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.164619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.161503    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:13:45.203606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.186127    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.211568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.211031    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.498040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.497947    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:13:45.530711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.529856    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:13:45.772935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.772849    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:13:45.814402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.812676    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.903171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.902842    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:13:45.904423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.904365    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.920995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.915390    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.947554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.919843    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:13:45.947554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.919893    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:13:45.947554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.928417    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.947554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.928746    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:13:45.947554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.935960    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:13:45.988612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.984009    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:13:45.988612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.987492    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:13:45.992321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.992297    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:13:45.992640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.992626    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:13:45.992822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.992797    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:13:45.996212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.996193    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:13:45.998816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:45.998799    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:13:46.013856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:46.013833    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:13:46.019362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:46.019337    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:13:47.788163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.788077    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:13:47.810074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.810043    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:13:47.810240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.810177    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.813860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.813829    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8da3b70c-openshift-default-dockercfg-njp2x\""
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.926753    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pds\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.926845    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.926897    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.926990    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.927031    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.927116    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.927171    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.927221    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:47.927593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:47.927263    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.028882    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.028956    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029000    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029039    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029167    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pds\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029234    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029270    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029357    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.029557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.029396    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.040627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.030316    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.068442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.068363    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.068638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.068449    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.068999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.068978    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.069756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.069734    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.106557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.093816    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pds\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.123231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.123190    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.144557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.144124    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.150788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.150721    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\") pod \"e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") " pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.425374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.425342    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:48.750805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.750761    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:48.751004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.750856    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:48.751004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.750904    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:48.785998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:48.785332    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:13:49.656975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:49.656936    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" event={"ID":"55737394-a6d7-4d24-9eb9-11a329fafdd9","Type":"ContainerStarted","Data":"6e5221886cc56e0fb3d9684c37c19dafc2d74e1316758c0cdf874e24f52f77be"}
Nov 02 22:13:49.657442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:49.657018    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" event={"ID":"55737394-a6d7-4d24-9eb9-11a329fafdd9","Type":"ContainerStarted","Data":"770fe1e2a0e1327a20ef53325980c379ca9e4745e16f1b8d46aa6d70a3977850"}
Nov 02 22:13:49.678821 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:49.678773    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" podStartSLOduration=2.678759336 podStartE2EDuration="2.678759336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:49.676585469 +0000 UTC m=+966.984676279" watchObservedRunningTime="2026-11-02 22:13:49.678759336 +0000 UTC m=+966.986850206"
Nov 02 22:13:50.425620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.425582    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:50.431141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.431115    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:50.695106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.695029    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:50.696169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.696145    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:13:50.819136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.819101    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:13:50.824633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.824603    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.831960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.831927    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d47f8605-openshift-default-dockercfg-qfzcv\""
Nov 02 22:13:50.843037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.843008    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:13:50.844616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844312    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844362    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844424    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhg9\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844461    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844632    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844671    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844701    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844774    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:50.844990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.844888    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.953861    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954017    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954052    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954111    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhg9\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954148    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954246    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954280    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954313    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.954372    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.958288    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.959026    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.981918    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.982447    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.995124    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhg9\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:50.996838    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.013403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.008099    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.038205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.023215    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.038827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.038760    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\") pod \"e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") " pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.172556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.166258    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:51.720789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.720759    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:13:51.721111 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:51.721041    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1907a5_a28e_4cff_a97a_9cc9c1d121d4.slice/crio-ada81c5c4de4a94b1d6813ae9a4d121b785ebb585d25c9bd6b0ba395cc783fed WatchSource:0}: Error finding container ada81c5c4de4a94b1d6813ae9a4d121b785ebb585d25c9bd6b0ba395cc783fed: Status 404 returned error can't find the container with id ada81c5c4de4a94b1d6813ae9a4d121b785ebb585d25c9bd6b0ba395cc783fed
Nov 02 22:13:51.723767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.723732    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:51.723827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.723806    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:51.723863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:51.723835    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:52.608325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.608292    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:13:52.613002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.612981    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.618941 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.618911    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8da3b70c\""
Nov 02 22:13:52.619587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.619508    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8da3b70c-dockercfg-2s2g6\""
Nov 02 22:13:52.619587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.619523    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:13:52.667548 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.667502    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:13:52.672041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.672023    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.681813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.681787    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:13:52.751568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.750413    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.751568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.750887    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.751568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.751050    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcx4h\" (UniqueName: \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.751568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.751130    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8bz\" (UniqueName: \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.869143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.866415    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.869143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.866562    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcx4h\" (UniqueName: \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.869143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.866639    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8bz\" (UniqueName: \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.869143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.866700    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.869143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.867695    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:52.873861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.873326    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:52.969567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:52.921292    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8bz\" (UniqueName: \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\") pod \"payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") " pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:53.043176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.005263    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:53.114073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.047160    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcx4h\" (UniqueName: \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\") pod \"payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:53.114073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.057020    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" event={"ID":"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4","Type":"ContainerStarted","Data":"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"}
Nov 02 22:13:53.114073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.090040    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" event={"ID":"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4","Type":"ContainerStarted","Data":"ada81c5c4de4a94b1d6813ae9a4d121b785ebb585d25c9bd6b0ba395cc783fed"}
Nov 02 22:13:53.114073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.083524    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" podStartSLOduration=3.083504127 podStartE2EDuration="3.083504127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:53.083017977 +0000 UTC m=+970.391108818" watchObservedRunningTime="2026-11-02 22:13:53.083504127 +0000 UTC m=+970.391594936"
Nov 02 22:13:53.171440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.171403    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:53.182687 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.182653    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:53.236508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.236467    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:53.531857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.531829    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:13:53.533457 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:53.533428    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2691196b_2823_42fe_8bb0_cbbf0b0cc642.slice/crio-f1ddcd4367f7a55670e81e414c30b56ccd23448529e5a578c969e07febb766b4 WatchSource:0}: Error finding container f1ddcd4367f7a55670e81e414c30b56ccd23448529e5a578c969e07febb766b4: Status 404 returned error can't find the container with id f1ddcd4367f7a55670e81e414c30b56ccd23448529e5a578c969e07febb766b4
Nov 02 22:13:53.676416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:53.676386    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:13:53.676605 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:13:53.676582    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b0694b_1fae_482e_8548_6b034a223ce0.slice/crio-64679b9224ace8e1f2cad2e54a55e44a3997fb7724064e8be6efb9212e9591eb WatchSource:0}: Error finding container 64679b9224ace8e1f2cad2e54a55e44a3997fb7724064e8be6efb9212e9591eb: Status 404 returned error can't find the container with id 64679b9224ace8e1f2cad2e54a55e44a3997fb7724064e8be6efb9212e9591eb
Nov 02 22:13:54.064443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.064354    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" event={"ID":"b0b0694b-1fae-482e-8548-6b034a223ce0","Type":"ContainerStarted","Data":"8be1373f423006b26de358ab53bea077a3d87dcf11e7aec739b320d1e2c26d43"}
Nov 02 22:13:54.064443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.064417    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" event={"ID":"b0b0694b-1fae-482e-8548-6b034a223ce0","Type":"ContainerStarted","Data":"64679b9224ace8e1f2cad2e54a55e44a3997fb7724064e8be6efb9212e9591eb"}
Nov 02 22:13:54.064895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.064561    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:13:54.066198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.066175    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" event={"ID":"2691196b-2823-42fe-8bb0-cbbf0b0cc642","Type":"ContainerStarted","Data":"8cf20ffd8114cac8ab6df95f1a323f89082859705db7737073051c8e82965f79"}
Nov 02 22:13:54.066294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.066220    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" event={"ID":"2691196b-2823-42fe-8bb0-cbbf0b0cc642","Type":"ContainerStarted","Data":"f1ddcd4367f7a55670e81e414c30b56ccd23448529e5a578c969e07febb766b4"}
Nov 02 22:13:54.066294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.066258    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:54.066294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.066289    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:13:54.067160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.067143    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:13:54.080931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.080879    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" podStartSLOduration=2.0808629 podStartE2EDuration="2.0808629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:54.078350732 +0000 UTC m=+971.386441539" watchObservedRunningTime="2026-11-02 22:13:54.0808629 +0000 UTC m=+971.388953706"
Nov 02 22:13:54.101280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:54.101232    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" podStartSLOduration=2.10121941 podStartE2EDuration="2.10121941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:54.09642561 +0000 UTC m=+971.404516427" watchObservedRunningTime="2026-11-02 22:13:54.10121941 +0000 UTC m=+971.409310214"
Nov 02 22:13:55.310516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.310478    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:13:55.316691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.316664    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.334584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.333203    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:13:55.348143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.348114    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6a72dd63-openshift-default-dockercfg-h5zk5\""
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.467682    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwrl\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.467816    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.467902    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.467946    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.467974    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.468012    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.468042    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.468140    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.472553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.468246    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.572862    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573066    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwrl\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573301    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573444    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573500    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573575    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573628    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573666    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.579611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.573791    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.588557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.588243    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.588557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.588509    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.588803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.588776    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.594555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.589249    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.594555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.590077    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.617182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.616381    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.654823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.652036    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.656637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.655133    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.665694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.658242    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwrl\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\") pod \"e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") " pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.946814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.946777    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:55.959222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.959002    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:13:55.963990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.963881    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:55.989513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:55.981659    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:13:56.071559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.066272    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d47f8605-dockercfg-nd8fd\""
Nov 02 22:13:56.071559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.066370    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d47f8605\""
Nov 02 22:13:56.215607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.202164    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.215607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.202555    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.305751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.303671    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.305751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.303914    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.305751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.305191    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.378036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.377967    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\") pod \"payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") " pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:56.411636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.411301    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:57.003649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:56.985766    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:13:57.225429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:57.225389    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:13:57.234731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:57.234631    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:57.234731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:57.234732    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:57.234990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:57.234776    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:13:57.237646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:57.237579    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" event={"ID":"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5","Type":"ContainerStarted","Data":"726c3c4f5b80318cbd5edf3ebb877b50e53174a40a299900f6f3fa92566b67ef"}
Nov 02 22:13:57.505577 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:13:57.505515    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:13:58.247521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.247486    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" event={"ID":"f238a134-f788-4d64-a534-a682aea57409","Type":"ContainerStarted","Data":"ccf9e0162c8871fb247be9e2eb912532358591b1a83f086a316d5b2b73bb6ab7"}
Nov 02 22:13:58.247783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.247762    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:13:58.247846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.247812    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" event={"ID":"f238a134-f788-4d64-a534-a682aea57409","Type":"ContainerStarted","Data":"5f4452a647cbbbc5880c41da0a387ecc688c0a7cb5d80f63883cc34414ff72a1"}
Nov 02 22:13:58.249410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.249388    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" event={"ID":"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5","Type":"ContainerStarted","Data":"8a52660052e57d497aef683a0054dfe304696905690a5f403bd8e2d495f88918"}
Nov 02 22:13:58.265557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.265498    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" podStartSLOduration=3.265486707 podStartE2EDuration="3.265486707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:58.263905485 +0000 UTC m=+975.571996289" watchObservedRunningTime="2026-11-02 22:13:58.265486707 +0000 UTC m=+975.573577545"
Nov 02 22:13:58.286245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.286196    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" podStartSLOduration=3.286184376 podStartE2EDuration="3.286184376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:58.285375055 +0000 UTC m=+975.593465857" watchObservedRunningTime="2026-11-02 22:13:58.286184376 +0000 UTC m=+975.594275179"
Nov 02 22:13:58.947461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.947420    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:58.953720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:58.953694    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:59.257620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:59.257520    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:13:59.258508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:13:59.258487    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:14:00.951130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:00.951087    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:14:01.669428 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.669388    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:01.745341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.676723    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:01.745341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.684077    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:01.796605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.764222    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6a72dd63\""
Nov 02 22:14:01.796880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.768964    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6a72dd63-dockercfg-hwxxs\""
Nov 02 22:14:01.798335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.769033    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:01.800902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.800872    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:01.801065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.801033    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhmw\" (UniqueName: \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:01.802668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.802600    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:01.985471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.971039    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:01.985471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.971251    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhmw\" (UniqueName: \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:01.985471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:01.973056    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:02.119559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.119129    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhmw\" (UniqueName: \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\") pod \"payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:02.368815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.368644    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:02.675248 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.675168    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:14:02.686289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.686259    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:02.694305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.694018    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-10852f9c\""
Nov 02 22:14:02.694305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.694175    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-10852f9c-dockercfg-dqtwf\""
Nov 02 22:14:02.696843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.696815    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:14:02.825416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.737675    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:14:02.907795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.851031    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:02.907795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.851258    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqr47\" (UniqueName: \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.914294    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.954960    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.955053    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.955091    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.955272    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqr47\" (UniqueName: \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:02.977243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:02.957099    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:03.061718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.051414    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqr47\" (UniqueName: \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\") pod \"payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:03.106373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.066695    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:03.106373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.066773    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:03.106373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.069122    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:03.119295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.119117    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:14:03.256376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.179162    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\") pod \"payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") " pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:03.392562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.357479    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:03.395884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.394753    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:03.450357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.448979    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:03.706558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.706082    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" event={"ID":"61816091-d007-4af4-90b6-52cfbe8ea50b","Type":"ContainerStarted","Data":"93ed018554e248f2dead4cb010debaba72b19854dabfc4b7c7151a9e35850096"}
Nov 02 22:14:03.836616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.832418    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:14:03.836616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:03.832751    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerName="istio-proxy" containerID="cri-o://36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b" gracePeriod=30
Nov 02 22:14:04.574624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.565438    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:14:04.574624 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:14:04.567493    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e415847_630d_47ee_8cda_523e1453744c.slice/crio-6cb469212fde30008a6a7916714e98ee6890b2323874736e42b1ae6db95a6e9f WatchSource:0}: Error finding container 6cb469212fde30008a6a7916714e98ee6890b2323874736e42b1ae6db95a6e9f: Status 404 returned error can't find the container with id 6cb469212fde30008a6a7916714e98ee6890b2323874736e42b1ae6db95a6e9f
Nov 02 22:14:04.577964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.577935    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:14:04.588703 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:14:04.588673    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e6dca8_b3f2_4c2d_97bf_e50c207e5e81.slice/crio-8e16158a946a866f91756d6c79e708bfb59d07155a7e6560cdff36af49b858a2 WatchSource:0}: Error finding container 8e16158a946a866f91756d6c79e708bfb59d07155a7e6560cdff36af49b858a2: Status 404 returned error can't find the container with id 8e16158a946a866f91756d6c79e708bfb59d07155a7e6560cdff36af49b858a2
Nov 02 22:14:04.776255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.768826    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" event={"ID":"0e415847-630d-47ee-8cda-523e1453744c","Type":"ContainerStarted","Data":"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"}
Nov 02 22:14:04.776255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.768905    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" event={"ID":"0e415847-630d-47ee-8cda-523e1453744c","Type":"ContainerStarted","Data":"6cb469212fde30008a6a7916714e98ee6890b2323874736e42b1ae6db95a6e9f"}
Nov 02 22:14:04.776255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.768980    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:04.816436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.816352    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" podStartSLOduration=2.816312501 podStartE2EDuration="2.816312501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:04.800187907 +0000 UTC m=+982.108278714" watchObservedRunningTime="2026-11-02 22:14:04.816312501 +0000 UTC m=+982.124403327"
Nov 02 22:14:04.907117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.903305    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:04.907117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.903369    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:04.907117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.903396    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" event={"ID":"61816091-d007-4af4-90b6-52cfbe8ea50b","Type":"ContainerStarted","Data":"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"}
Nov 02 22:14:04.907117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.903427    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" event={"ID":"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81","Type":"ContainerStarted","Data":"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"}
Nov 02 22:14:04.907117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.903447    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" event={"ID":"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81","Type":"ContainerStarted","Data":"8e16158a946a866f91756d6c79e708bfb59d07155a7e6560cdff36af49b858a2"}
Nov 02 22:14:04.915017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.910441    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" podStartSLOduration=3.91042261 podStartE2EDuration="3.91042261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:04.864111126 +0000 UTC m=+982.172202000" watchObservedRunningTime="2026-11-02 22:14:04.91042261 +0000 UTC m=+982.218513416"
Nov 02 22:14:04.926570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:04.926475    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" podStartSLOduration=2.926453859 podStartE2EDuration="2.926453859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:04.919840391 +0000 UTC m=+982.227931197" watchObservedRunningTime="2026-11-02 22:14:04.926453859 +0000 UTC m=+982.234544665"
Nov 02 22:14:05.088935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:05.088900    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:14:05.089317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:05.089295    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:14:05.964555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:05.953652    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" podUID="91e1feec-0fc3-401e-aff6-ba2d507887bd" containerName="payload-pre-processing" containerID="cri-o://c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7" gracePeriod=25
Nov 02 22:14:06.426169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.426143    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:14:06.434175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.434146    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\") pod \"91e1feec-0fc3-401e-aff6-ba2d507887bd\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") "
Nov 02 22:14:06.434317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.434236    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\" (UniqueName: \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\") pod \"91e1feec-0fc3-401e-aff6-ba2d507887bd\" (UID: \"91e1feec-0fc3-401e-aff6-ba2d507887bd\") "
Nov 02 22:14:06.434569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.434514    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume" pod "91e1feec-0fc3-401e-aff6-ba2d507887bd" (UID: "91e1feec-0fc3-401e-aff6-ba2d507887bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:06.434736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.434722    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e1feec-0fc3-401e-aff6-ba2d507887bd-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:06.436266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.436243    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w" pod "91e1feec-0fc3-401e-aff6-ba2d507887bd" (UID: "91e1feec-0fc3-401e-aff6-ba2d507887bd"). InnerVolumeSpecName "kube-api-access-8wx2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:06.536716 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:06.536634    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wx2w\" (UniqueName: \"kubernetes.io/projected/91e1feec-0fc3-401e-aff6-ba2d507887bd-kube-api-access-8wx2w\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:07.008557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.005509    4929 generic.go:356] "Generic (PLEG): container finished" podID="91e1feec-0fc3-401e-aff6-ba2d507887bd" containerID="c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7" exitCode=0
Nov 02 22:14:07.008557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.006073    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" event={"ID":"91e1feec-0fc3-401e-aff6-ba2d507887bd","Type":"ContainerDied","Data":"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"}
Nov 02 22:14:07.008557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.006135    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9" event={"ID":"91e1feec-0fc3-401e-aff6-ba2d507887bd","Type":"ContainerDied","Data":"6205e6fe3fafbee8135cf20e2342ec04a6e28d4791e01e43a827588429590801"}
Nov 02 22:14:07.008557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.006166    4929 scope.go:122] "RemoveContainer" containerID="c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"
Nov 02 22:14:07.008557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.006681    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"
Nov 02 22:14:07.050557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.049884    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:14:07.072351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.061095    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec62fcf4-7646b795b7-vg6m9"]
Nov 02 22:14:07.094402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.094378    4929 scope.go:122] "RemoveContainer" containerID="c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"
Nov 02 22:14:07.094981 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:07.094954    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7\": container with ID starting with c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7 not found: ID does not exist" containerID="c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"
Nov 02 22:14:07.095158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:07.095129    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7"} err="failed to get container status \"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7\": rpc error: code = NotFound desc = could not find container \"c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7\": container with ID starting with c328360e3083f49fa284a14a02f48e5684b632ca84339352c6aaac4917e26df7 not found: ID does not exist"
Nov 02 22:14:08.605121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.605089    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:14:08.605939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.605920    4929 cpu_manager.go:414] "containerMap: removing container" podUID="91e1feec-0fc3-401e-aff6-ba2d507887bd" containerName="payload-pre-processing"
Nov 02 22:14:08.605939 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.605939    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1feec-0fc3-401e-aff6-ba2d507887bd" containerName="payload-pre-processing"
Nov 02 22:14:08.606064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.606057    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1feec-0fc3-401e-aff6-ba2d507887bd" containerName="payload-pre-processing"
Nov 02 22:14:08.613434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.613294    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.621030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.619224    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:14:08.621030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.619284    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bb16f94c\""
Nov 02 22:14:08.621030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.620049    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bb16f94c-dockercfg-trs7t\""
Nov 02 22:14:08.650871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.650840    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:08.656776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.656743    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.668396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.668368    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:08.707081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.707031    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fjn\" (UniqueName: \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.707281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.707120    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.707281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.707255    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.707490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.707446    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldfh\" (UniqueName: \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.848555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.840729    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldfh\" (UniqueName: \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.848555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.841593    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fjn\" (UniqueName: \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.848555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.848220    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.871643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.849159    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:08.875135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.873830    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.875135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.875081    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:08.978173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:08.960249    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fjn\" (UniqueName: \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\") pod \"payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:09.033066 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.032764    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldfh\" (UniqueName: \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\") pod \"payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:09.069222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.069183    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e1feec-0fc3-401e-aff6-ba2d507887bd" path="/var/lib/kubelet/pods/91e1feec-0fc3-401e-aff6-ba2d507887bd/volumes"
Nov 02 22:14:09.273613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.267062    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:09.318647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.290563    4929 patch_prober.go:29] interesting pod/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 22:14:09.318647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.290623    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 22:14:09.318647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.291763    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:09.345341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.343882    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:14:09.563059 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.563024    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:14:09.624869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.624836    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.624885    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.624909    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.624986    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.625171    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.625210    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.625254    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.625282    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.625722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.625365    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\") pod \"a1652513-8ac1-4447-92f2-b8c93430ccbd\" (UID: \"a1652513-8ac1-4447-92f2-b8c93430ccbd\") "
Nov 02 22:14:09.626156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626123    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:09.626156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626143    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:09.626386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626355    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:09.626870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626828    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.626870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626856    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.627027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.626871    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.627155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.627126    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:09.627283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.627200    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:09.628932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.628905    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:09.629027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.628953    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:09.629027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.628979    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "kube-api-access-hllms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:09.629811 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.629786    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo" pod "a1652513-8ac1-4447-92f2-b8c93430ccbd" (UID: "a1652513-8ac1-4447-92f2-b8c93430ccbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.727490    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.731988    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.732021    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.732038    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.732051    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1652513-8ac1-4447-92f2-b8c93430ccbd-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.732064    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1652513-8ac1-4447-92f2-b8c93430ccbd-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.734700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.732078    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/a1652513-8ac1-4447-92f2-b8c93430ccbd-kube-api-access-hllms\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:09.754001 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:14:09.753965    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded49ca6e_3508_403d_bd18_8959890821c6.slice/crio-9090cbaacc9d9b31db5de45d93c37b62216c8ee2b5ae6d19fd27c42d22028f7b WatchSource:0}: Error finding container 9090cbaacc9d9b31db5de45d93c37b62216c8ee2b5ae6d19fd27c42d22028f7b: Status 404 returned error can't find the container with id 9090cbaacc9d9b31db5de45d93c37b62216c8ee2b5ae6d19fd27c42d22028f7b
Nov 02 22:14:09.782486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:09.782451    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:10.060162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.060068    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" event={"ID":"ed49ca6e-3508-403d-bd18-8959890821c6","Type":"ContainerStarted","Data":"b37211911718d1c3370936e5a94bff02b33ff2aa5d350efb186e5f3a0b3452d1"}
Nov 02 22:14:10.060381 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.060172    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:10.060381 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.060201    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" event={"ID":"ed49ca6e-3508-403d-bd18-8959890821c6","Type":"ContainerStarted","Data":"9090cbaacc9d9b31db5de45d93c37b62216c8ee2b5ae6d19fd27c42d22028f7b"}
Nov 02 22:14:10.061651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.061620    4929 generic.go:356] "Generic (PLEG): container finished" podID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerID="36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b" exitCode=0
Nov 02 22:14:10.061767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.061675    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"
Nov 02 22:14:10.061767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.061692    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" event={"ID":"a1652513-8ac1-4447-92f2-b8c93430ccbd","Type":"ContainerDied","Data":"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"}
Nov 02 22:14:10.061767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.061731    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb" event={"ID":"a1652513-8ac1-4447-92f2-b8c93430ccbd","Type":"ContainerDied","Data":"f284fa6cebbcfa91a617f9a2a984a289c11944234000909f7e4e00e1ad20d2f7"}
Nov 02 22:14:10.061767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.061760    4929 scope.go:122] "RemoveContainer" containerID="36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"
Nov 02 22:14:10.067611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.067586    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" event={"ID":"49ffd9f9-0e2f-4168-aff9-e2a5a014248b","Type":"ContainerStarted","Data":"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"}
Nov 02 22:14:10.067735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.067629    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" event={"ID":"49ffd9f9-0e2f-4168-aff9-e2a5a014248b","Type":"ContainerStarted","Data":"342a27dcb1247abd3a22aa2287a171a10d9e90b271a22c21970a84b0e9aa7d81"}
Nov 02 22:14:10.067735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.067662    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:10.072918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.072901    4929 scope.go:122] "RemoveContainer" containerID="36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"
Nov 02 22:14:10.073149 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:10.073132    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b\": container with ID starting with 36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b not found: ID does not exist" containerID="36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"
Nov 02 22:14:10.073194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.073156    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b"} err="failed to get container status \"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b\": rpc error: code = NotFound desc = could not find container \"36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b\": container with ID starting with 36c264d585c3f48c9a348667da4df2035c2bed06ff9bfa9b64a9803d613b735b not found: ID does not exist"
Nov 02 22:14:10.079101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.079046    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" podStartSLOduration=2.079032659 podStartE2EDuration="2.079032659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:10.078657878 +0000 UTC m=+987.386748694" watchObservedRunningTime="2026-11-02 22:14:10.079032659 +0000 UTC m=+987.387123465"
Nov 02 22:14:10.098450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.098410    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" podStartSLOduration=2.098397973 podStartE2EDuration="2.098397973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:10.0940089 +0000 UTC m=+987.402099706" watchObservedRunningTime="2026-11-02 22:14:10.098397973 +0000 UTC m=+987.406488794"
Nov 02 22:14:10.109374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.109343    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:14:10.113048 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.113025    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ec62fcf4-openshift-default-5f494c969f-fznwb"]
Nov 02 22:14:10.806498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:10.806465    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" path="/var/lib/kubelet/pods/a1652513-8ac1-4447-92f2-b8c93430ccbd/volumes"
Nov 02 22:14:15.965211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:15.963761    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:14:15.989854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:15.989822    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:16.803560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:16.803120    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 02 22:14:16.938559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:16.938122    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:14:16.968562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:16.947619    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.016949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.016906    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.019324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.017192    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.046807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.046772    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:14:17.255709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.255657    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:14:17.362446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.362398    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.378606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.362910    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.507772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.507661    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.574985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.574939    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.813711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.804175    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.942708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.942649    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:17.944079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.944046    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:14:17.944601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.944581    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:14:17.946717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.945610    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.964590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.964527    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:17.965260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:17.965236    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:14:18.163893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.154941    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.163893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.155474    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:14:18.163893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.157172    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:18.163893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.157652    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:14:18.426615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.365156    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.473597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.382668    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.474225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.407606    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:14:18.486562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.431140    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:14:18.486562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.475942    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:14:18.486562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.476003    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:14:18.486562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.431690    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.543564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.508021    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.666612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.583656    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:14:18.738611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.714666    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.738611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.715292    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:14:18.798559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.796359    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 22:14:18.993717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.956057    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:14:18.993717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.957104    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.993717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.957983    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:18.993717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.958803    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:18.993717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:18.970861    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:19.114001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.113953    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:19.114310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.114274    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:19.261623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.260054    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.295571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.286699    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.383557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.360054    4929 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:19.383557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.360144    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:19.463147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.447810    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:19.476620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.474266    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:19.483671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.483623    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.594357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.594249    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.729556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.727072    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.829558 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:19.821814    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.024s"
Nov 02 22:14:19.829951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.829923    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:14:19.830139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.830125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:14:19.830265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.830253    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:14:19.962755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.962708    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.964495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.964455    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:14:19.977681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.977621    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:14:20.013708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.986703    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:20.019558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:19.988840    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:14:20.314611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.313682    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:14:20.321580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.315154    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:20.462879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.462823    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:20.487691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.487636    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:20.489219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.489182    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:20.702318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.702269    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:20.739521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.737697    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:20.865644 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:20.809687    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:21.119650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.104209    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.119650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.104622    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.119650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.110043    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.149605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.123776    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.149605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.124099    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.230593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.222623    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:21.230593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.222788    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:21.247568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.247498    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:21.415622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.401330    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.416436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.412622    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.471243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.470174    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:14:21.502560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.500642    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:21.502560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.501073    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.038715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.999048    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.038715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:21.999891    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.162558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.105464    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.162558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.159572    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.162558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.162275    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:22.187444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.187370    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:22.423965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.122817    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.482568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.456459    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:22.482914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.482875    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:22.483462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.483433    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.488045 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.488014    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:14:22.570562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.536851    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:22.571133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:22.561097    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:14:23.007729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.007678    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.019556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.008271    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.058073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.019846    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:23.122808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.122765    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.123637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.106161    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:23.124148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.057713    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:23.152413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.147309    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.161210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.161135    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.166026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.165982    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.218680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.218632    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.218967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.218930    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:14:23.477287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.477237    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.489327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.488646    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.489327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.489034    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:23.520678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.519636    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.584651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.583643    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.584651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.584226    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:14:23.604663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.604624    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:23.604915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.604883    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:23.771620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.768920    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:14:23.771620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:23.768985    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.042616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.040478    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.106856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.082379    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.106856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.097832    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.164601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.152074    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.195339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.184173    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.195339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.185257    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.234953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.234897    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.509888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.509832    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.553375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.527788    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.553375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.545304    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.553375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.546829    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.641569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.591696    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:24.641569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.592464    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:14:24.643418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:24.643371    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.080634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.068118    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.105568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.082496    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.106391 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.096937    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:14:25.199558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.191094    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.222572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.222510    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.431126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.431080    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.450848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.450807    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Nov 02 22:14:25.525551 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.521813    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.542556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.538736    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.569563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.561746    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.569563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.562113    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.610363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.610315    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:25.610948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.610915    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:14:25.944557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:25.936623    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:14:26.091179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.083364    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:14:26.126242 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.126209    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:14:26.149249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.149157    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:14:26.154838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.154812    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:14:26.186728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.186696    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:14:26.204176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.204074    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:14:26.208214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.186995    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:14:26.229630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.227772    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:14:26.234793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.232628    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:14:26.241338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.241313    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:14:26.505606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:26.496938    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:14:27.498512 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:27.498480    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:14:36.446755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:36.446719    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:14:36.457171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:36.457141    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:14:41.447626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.447577    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" podUID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" containerName="payload-processing" containerID="cri-o://8cf20ffd8114cac8ab6df95f1a323f89082859705db7737073051c8e82965f79" gracePeriod=25
Nov 02 22:14:41.458158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.458116    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" podUID="b0b0694b-1fae-482e-8548-6b034a223ce0" containerName="payload-pre-processing" containerID="cri-o://8be1373f423006b26de358ab53bea077a3d87dcf11e7aec739b320d1e2c26d43" gracePeriod=25
Nov 02 22:14:41.568293 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.568264    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:14:41.590373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.590341    4929 generic.go:356] "Generic (PLEG): container finished" podID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" containerID="8cf20ffd8114cac8ab6df95f1a323f89082859705db7737073051c8e82965f79" exitCode=0
Nov 02 22:14:41.590560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.590440    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" event={"ID":"2691196b-2823-42fe-8bb0-cbbf0b0cc642","Type":"ContainerDied","Data":"8cf20ffd8114cac8ab6df95f1a323f89082859705db7737073051c8e82965f79"}
Nov 02 22:14:41.591961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.591932    4929 generic.go:356] "Generic (PLEG): container finished" podID="b0b0694b-1fae-482e-8548-6b034a223ce0" containerID="8be1373f423006b26de358ab53bea077a3d87dcf11e7aec739b320d1e2c26d43" exitCode=0
Nov 02 22:14:41.592089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:41.591971    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" event={"ID":"b0b0694b-1fae-482e-8548-6b034a223ce0","Type":"ContainerDied","Data":"8be1373f423006b26de358ab53bea077a3d87dcf11e7aec739b320d1e2c26d43"}
Nov 02 22:14:42.026424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.025738    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:14:42.180629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.179192    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\" (UniqueName: \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\") pod \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") "
Nov 02 22:14:42.180629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.179500    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\") pod \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\" (UID: \"2691196b-2823-42fe-8bb0-cbbf0b0cc642\") "
Nov 02 22:14:42.181746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.181032    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:14:42.204880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.204763    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume" pod "2691196b-2823-42fe-8bb0-cbbf0b0cc642" (UID: "2691196b-2823-42fe-8bb0-cbbf0b0cc642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:42.230625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.230526    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz" pod "2691196b-2823-42fe-8bb0-cbbf0b0cc642" (UID: "2691196b-2823-42fe-8bb0-cbbf0b0cc642"). InnerVolumeSpecName "kube-api-access-gz8bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:42.281576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.281456    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\") pod \"b0b0694b-1fae-482e-8548-6b034a223ce0\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") "
Nov 02 22:14:42.281974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.281952    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\" (UniqueName: \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\") pod \"b0b0694b-1fae-482e-8548-6b034a223ce0\" (UID: \"b0b0694b-1fae-482e-8548-6b034a223ce0\") "
Nov 02 22:14:42.282971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.282948    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2691196b-2823-42fe-8bb0-cbbf0b0cc642-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:42.283112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.283096    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz8bz\" (UniqueName: \"kubernetes.io/projected/2691196b-2823-42fe-8bb0-cbbf0b0cc642-kube-api-access-gz8bz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:42.284000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.283957    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume" pod "b0b0694b-1fae-482e-8548-6b034a223ce0" (UID: "b0b0694b-1fae-482e-8548-6b034a223ce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:42.289463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.289429    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h" pod "b0b0694b-1fae-482e-8548-6b034a223ce0" (UID: "b0b0694b-1fae-482e-8548-6b034a223ce0"). InnerVolumeSpecName "kube-api-access-xcx4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:42.385335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.385288    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b0694b-1fae-482e-8548-6b034a223ce0-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:42.385335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.385330    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcx4h\" (UniqueName: \"kubernetes.io/projected/b0b0694b-1fae-482e-8548-6b034a223ce0-kube-api-access-xcx4h\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:42.659591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.657799    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5" event={"ID":"2691196b-2823-42fe-8bb0-cbbf0b0cc642","Type":"ContainerDied","Data":"f1ddcd4367f7a55670e81e414c30b56ccd23448529e5a578c969e07febb766b4"}
Nov 02 22:14:42.659591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.657879    4929 scope.go:122] "RemoveContainer" containerID="8cf20ffd8114cac8ab6df95f1a323f89082859705db7737073051c8e82965f79"
Nov 02 22:14:42.659591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.658024    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"
Nov 02 22:14:42.706007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.705977    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf" event={"ID":"b0b0694b-1fae-482e-8548-6b034a223ce0","Type":"ContainerDied","Data":"64679b9224ace8e1f2cad2e54a55e44a3997fb7724064e8be6efb9212e9591eb"}
Nov 02 22:14:42.706297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.706280    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"
Nov 02 22:14:42.774911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.774463    4929 scope.go:122] "RemoveContainer" containerID="8be1373f423006b26de358ab53bea077a3d87dcf11e7aec739b320d1e2c26d43"
Nov 02 22:14:42.872852 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.872813    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:14:42.899434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.899331    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8da3b70c-5688654dd7-wtxx5"]
Nov 02 22:14:42.918516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.918377    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:14:42.925397 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.925357    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8da3b70c-689747964c-2jdlf"]
Nov 02 22:14:42.972911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.972878    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:14:42.973337 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:42.973307    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" podUID="55737394-a6d7-4d24-9eb9-11a329fafdd9" containerName="istio-proxy" containerID="cri-o://6e5221886cc56e0fb3d9684c37c19dafc2d74e1316758c0cdf874e24f52f77be" gracePeriod=30
Nov 02 22:14:44.867089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:44.867035    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" path="/var/lib/kubelet/pods/2691196b-2823-42fe-8bb0-cbbf0b0cc642/volumes"
Nov 02 22:14:44.868452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:44.868424    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b0694b-1fae-482e-8548-6b034a223ce0" path="/var/lib/kubelet/pods/b0b0694b-1fae-482e-8548-6b034a223ce0/volumes"
Nov 02 22:14:45.666372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:45.666230    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:14:45.667221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:45.666930    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" podUID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" containerName="istio-proxy" containerID="cri-o://9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3" gracePeriod=30
Nov 02 22:14:46.495806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:46.495772    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:46.568831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:46.568764    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" podUID="f238a134-f788-4d64-a534-a682aea57409" containerName="payload-processing" containerID="cri-o://ccf9e0162c8871fb247be9e2eb912532358591b1a83f086a316d5b2b73bb6ab7" gracePeriod=25
Nov 02 22:14:47.245600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.241615    4929 generic.go:356] "Generic (PLEG): container finished" podID="f238a134-f788-4d64-a534-a682aea57409" containerID="ccf9e0162c8871fb247be9e2eb912532358591b1a83f086a316d5b2b73bb6ab7" exitCode=0
Nov 02 22:14:47.245600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.241904    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" event={"ID":"f238a134-f788-4d64-a534-a682aea57409","Type":"ContainerDied","Data":"ccf9e0162c8871fb247be9e2eb912532358591b1a83f086a316d5b2b73bb6ab7"}
Nov 02 22:14:47.391705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.391653    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:14:47.399317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.399289    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400281    4929 cpu_manager.go:414] "containerMap: removing container" podUID="f238a134-f788-4d64-a534-a682aea57409" containerName="payload-processing"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400325    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f238a134-f788-4d64-a534-a682aea57409" containerName="payload-processing"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400347    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" containerName="payload-processing"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400355    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" containerName="payload-processing"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400409    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerName="istio-proxy"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400421    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerName="istio-proxy"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400500    4929 cpu_manager.go:414] "containerMap: removing container" podUID="b0b0694b-1fae-482e-8548-6b034a223ce0" containerName="payload-pre-processing"
Nov 02 22:14:47.400597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400510    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b0694b-1fae-482e-8548-6b034a223ce0" containerName="payload-pre-processing"
Nov 02 22:14:47.401076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400818    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="f238a134-f788-4d64-a534-a682aea57409" containerName="payload-processing"
Nov 02 22:14:47.401076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400836    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2691196b-2823-42fe-8bb0-cbbf0b0cc642" containerName="payload-processing"
Nov 02 22:14:47.401076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400866    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1652513-8ac1-4447-92f2-b8c93430ccbd" containerName="istio-proxy"
Nov 02 22:14:47.401076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.400886    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b0694b-1fae-482e-8548-6b034a223ce0" containerName="payload-pre-processing"
Nov 02 22:14:47.405566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.405528    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.407557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.407508    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:14:47.415820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.415752    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7144fe-openshift-default-dockercfg-p9x6x\""
Nov 02 22:14:47.536560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.536281    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\") pod \"f238a134-f788-4d64-a534-a682aea57409\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") "
Nov 02 22:14:47.536560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.536517    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\") pod \"f238a134-f788-4d64-a534-a682aea57409\" (UID: \"f238a134-f788-4d64-a534-a682aea57409\") "
Nov 02 22:14:47.537195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.536893    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537022    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537072    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537242    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537301    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537387    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537489    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537552    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.537627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.537605    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.573557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.569991    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume" pod "f238a134-f788-4d64-a534-a682aea57409" (UID: "f238a134-f788-4d64-a534-a682aea57409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:47.573557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.570301    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8" pod "f238a134-f788-4d64-a534-a682aea57409" (UID: "f238a134-f788-4d64-a534-a682aea57409"). InnerVolumeSpecName "kube-api-access-z6pm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:47.639874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.639836    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.640703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.640678    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.656498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.656463    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.656780 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.656757    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.656956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.656940    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.657186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.657172    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.657334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.657320    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.657657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.657632    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.701728    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.701967    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/f238a134-f788-4d64-a534-a682aea57409-kube-api-access-z6pm8\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.701989    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f238a134-f788-4d64-a534-a682aea57409-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.689310    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.689464    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.640505    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.702369    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.664912    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.686572    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.705095    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.712443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.709069    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:47.750108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:47.750047    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\") pod \"e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") " pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:48.034604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.030791    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:48.253171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.253087    4929 generic.go:356] "Generic (PLEG): container finished" podID="55737394-a6d7-4d24-9eb9-11a329fafdd9" containerID="6e5221886cc56e0fb3d9684c37c19dafc2d74e1316758c0cdf874e24f52f77be" exitCode=0
Nov 02 22:14:48.253343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.253165    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" event={"ID":"55737394-a6d7-4d24-9eb9-11a329fafdd9","Type":"ContainerDied","Data":"6e5221886cc56e0fb3d9684c37c19dafc2d74e1316758c0cdf874e24f52f77be"}
Nov 02 22:14:48.254718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.254693    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"
Nov 02 22:14:48.254846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.254699    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86" event={"ID":"f238a134-f788-4d64-a534-a682aea57409","Type":"ContainerDied","Data":"5f4452a647cbbbc5880c41da0a387ecc688c0a7cb5d80f63883cc34414ff72a1"}
Nov 02 22:14:48.254846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.254834    4929 scope.go:122] "RemoveContainer" containerID="ccf9e0162c8871fb247be9e2eb912532358591b1a83f086a316d5b2b73bb6ab7"
Nov 02 22:14:48.287722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.287689    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:14:48.294426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.294398    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d47f8605-7c68775c99-kwv86"]
Nov 02 22:14:48.381279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.381247    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:14:48.569205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.569168    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:14:48.571825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.571786    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:14:48.572049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.571873    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:14:48.572142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.572102    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:14:48.586321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586297    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.586452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586362    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.586452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586419    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.586580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586462    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.586656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586615    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:48.586942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586923    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.587040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.586983    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.587167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.587088    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.587229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.587179    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.587288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.587219    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:48.587288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.587250    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\") pod \"55737394-a6d7-4d24-9eb9-11a329fafdd9\" (UID: \"55737394-a6d7-4d24-9eb9-11a329fafdd9\") "
Nov 02 22:14:48.587802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.587780    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:48.588141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588115    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:48.588443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588417    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:48.588612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588591    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.588719 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588697    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.588719 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588719    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.588896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.588735    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.589657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.589625    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:48.589657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.589646    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:48.589772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.589700    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "kube-api-access-62pds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:48.589772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.589715    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo" pod "55737394-a6d7-4d24-9eb9-11a329fafdd9" (UID: "55737394-a6d7-4d24-9eb9-11a329fafdd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:14:48.690955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.690920    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.690955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.690956    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.691146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.690971    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55737394-a6d7-4d24-9eb9-11a329fafdd9-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.691146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.690982    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55737394-a6d7-4d24-9eb9-11a329fafdd9-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.691146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.690998    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pds\" (UniqueName: \"kubernetes.io/projected/55737394-a6d7-4d24-9eb9-11a329fafdd9-kube-api-access-62pds\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:48.810007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:48.809927    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f238a134-f788-4d64-a534-a682aea57409" path="/var/lib/kubelet/pods/f238a134-f788-4d64-a534-a682aea57409/volumes"
Nov 02 22:14:49.266222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.266182    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm" event={"ID":"55737394-a6d7-4d24-9eb9-11a329fafdd9","Type":"ContainerDied","Data":"770fe1e2a0e1327a20ef53325980c379ca9e4745e16f1b8d46aa6d70a3977850"}
Nov 02 22:14:49.266384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.266244    4929 scope.go:122] "RemoveContainer" containerID="6e5221886cc56e0fb3d9684c37c19dafc2d74e1316758c0cdf874e24f52f77be"
Nov 02 22:14:49.266384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.266345    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"
Nov 02 22:14:49.278441 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.278388    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerStarted","Data":"701d7efc37732cd8e4c19a3f4c199ffa945de7931a3fda379cc83c9212c68ebe"}
Nov 02 22:14:49.278609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.278502    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerStarted","Data":"6bc3fbc436aa41ac784971066a1b1d5585038e3d6289f1cb50260bdf0b3952e8"}
Nov 02 22:14:49.297505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.297472    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:14:49.301073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.301046    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8da3b70c-openshift-default-646c7b5d49-mscgm"]
Nov 02 22:14:49.314806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:49.314758    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podStartSLOduration=2.314744828 podStartE2EDuration="2.314744828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:49.310162769 +0000 UTC m=+1026.618253578" watchObservedRunningTime="2026-11-02 22:14:49.314744828 +0000 UTC m=+1026.622835633"
Nov 02 22:14:50.050927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.049614    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:50.074024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.073648    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:50.315607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.313285    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:50.328556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.321364    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:14:50.807405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.807371    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55737394-a6d7-4d24-9eb9-11a329fafdd9" path="/var/lib/kubelet/pods/55737394-a6d7-4d24-9eb9-11a329fafdd9/volumes"
Nov 02 22:14:50.963503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:50.963476    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:14:51.074353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074303    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074591    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074622    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074681    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074727    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074805    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074856    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.074895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074891    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.075244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074908    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:51.075244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.074985    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.075244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075087    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\") pod \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\" (UID: \"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4\") "
Nov 02 22:14:51.075244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075133    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:51.075425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075252    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:51.075425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075268    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:51.075779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075757    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.075898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075782    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.075898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075795    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.075898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075808    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.075898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.075821    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.077030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.077000    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:14:51.077221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.077199    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "kube-api-access-vwhg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:51.077456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.077421    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:51.077685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.077658    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy" pod "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" (UID: "ce1907a5-a28e-4cff-a97a-9cc9c1d121d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:51.181738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.178007    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.181738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.178052    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.181738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.178069    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.181738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.178085    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhg9\" (UniqueName: \"kubernetes.io/projected/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4-kube-api-access-vwhg9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:51.322906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.322870    4929 generic.go:356] "Generic (PLEG): container finished" podID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" containerID="9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3" exitCode=0
Nov 02 22:14:51.323108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.322917    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"
Nov 02 22:14:51.323108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.322963    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" event={"ID":"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4","Type":"ContainerDied","Data":"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"}
Nov 02 22:14:51.323108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.323024    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd" event={"ID":"ce1907a5-a28e-4cff-a97a-9cc9c1d121d4","Type":"ContainerDied","Data":"ada81c5c4de4a94b1d6813ae9a4d121b785ebb585d25c9bd6b0ba395cc783fed"}
Nov 02 22:14:51.323108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.323057    4929 scope.go:122] "RemoveContainer" containerID="9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"
Nov 02 22:14:51.334241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.334217    4929 scope.go:122] "RemoveContainer" containerID="9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"
Nov 02 22:14:51.334479 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:51.334465    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3\": container with ID starting with 9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3 not found: ID does not exist" containerID="9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"
Nov 02 22:14:51.334596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.334486    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3"} err="failed to get container status \"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3\": rpc error: code = NotFound desc = could not find container \"9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3\": container with ID starting with 9ef5249f2feaf2279bc659288d9cdd74fffbd162b4c02df9a5ab4b90085469b3 not found: ID does not exist"
Nov 02 22:14:51.348005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.347978    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:14:51.352343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.352321    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d47f8605-openshift-default-77847f9c45-7nxqd"]
Nov 02 22:14:51.496510 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.496428    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" podUID="61816091-d007-4af4-90b6-52cfbe8ea50b" containerName="payload-pre-processing" containerID="cri-o://8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d" gracePeriod=25
Nov 02 22:14:51.632108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.631162    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:14:51.632108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:51.631435    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" podUID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" containerName="istio-proxy" containerID="cri-o://8a52660052e57d497aef683a0054dfe304696905690a5f403bd8e2d495f88918" gracePeriod=30
Nov 02 22:14:52.172731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.172705    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:52.243587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.231944    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\" (UniqueName: \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\") pod \"61816091-d007-4af4-90b6-52cfbe8ea50b\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") "
Nov 02 22:14:52.243587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.233807    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\") pod \"61816091-d007-4af4-90b6-52cfbe8ea50b\" (UID: \"61816091-d007-4af4-90b6-52cfbe8ea50b\") "
Nov 02 22:14:52.243587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.236728    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume" pod "61816091-d007-4af4-90b6-52cfbe8ea50b" (UID: "61816091-d007-4af4-90b6-52cfbe8ea50b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:52.243587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.237245    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61816091-d007-4af4-90b6-52cfbe8ea50b-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:52.291894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.291847    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw" pod "61816091-d007-4af4-90b6-52cfbe8ea50b" (UID: "61816091-d007-4af4-90b6-52cfbe8ea50b"). InnerVolumeSpecName "kube-api-access-4xhmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:52.347581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.338938    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhmw\" (UniqueName: \"kubernetes.io/projected/61816091-d007-4af4-90b6-52cfbe8ea50b-kube-api-access-4xhmw\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:52.369568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.367485    4929 generic.go:356] "Generic (PLEG): container finished" podID="61816091-d007-4af4-90b6-52cfbe8ea50b" containerID="8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d" exitCode=0
Nov 02 22:14:52.369568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.367795    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"
Nov 02 22:14:52.369568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.368043    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" event={"ID":"61816091-d007-4af4-90b6-52cfbe8ea50b","Type":"ContainerDied","Data":"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"}
Nov 02 22:14:52.369568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.368119    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm" event={"ID":"61816091-d007-4af4-90b6-52cfbe8ea50b","Type":"ContainerDied","Data":"93ed018554e248f2dead4cb010debaba72b19854dabfc4b7c7151a9e35850096"}
Nov 02 22:14:52.369568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.368151    4929 scope.go:122] "RemoveContainer" containerID="8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"
Nov 02 22:14:52.395758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.395733    4929 scope.go:122] "RemoveContainer" containerID="8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"
Nov 02 22:14:52.396436 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:52.396403    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d\": container with ID starting with 8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d not found: ID does not exist" containerID="8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"
Nov 02 22:14:52.396625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.396597    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d"} err="failed to get container status \"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d\": rpc error: code = NotFound desc = could not find container \"8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d\": container with ID starting with 8782e8b339cddcf44cc23285c9ceb327236f186c4ba57aab6d8e725723a1a42d not found: ID does not exist"
Nov 02 22:14:52.401447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.401424    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:52.405190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.405170    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a72dd63-7b5cd687f5-8smsm"]
Nov 02 22:14:52.806775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.806701    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61816091-d007-4af4-90b6-52cfbe8ea50b" path="/var/lib/kubelet/pods/61816091-d007-4af4-90b6-52cfbe8ea50b/volumes"
Nov 02 22:14:52.807251 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.807236    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" path="/var/lib/kubelet/pods/ce1907a5-a28e-4cff-a97a-9cc9c1d121d4/volumes"
Nov 02 22:14:52.846688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.846653    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:14:52.847353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847338    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" containerName="istio-proxy"
Nov 02 22:14:52.847401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847355    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" containerName="istio-proxy"
Nov 02 22:14:52.847401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847371    4929 cpu_manager.go:414] "containerMap: removing container" podUID="61816091-d007-4af4-90b6-52cfbe8ea50b" containerName="payload-pre-processing"
Nov 02 22:14:52.847401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847377    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61816091-d007-4af4-90b6-52cfbe8ea50b" containerName="payload-pre-processing"
Nov 02 22:14:52.847401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847390    4929 cpu_manager.go:414] "containerMap: removing container" podUID="55737394-a6d7-4d24-9eb9-11a329fafdd9" containerName="istio-proxy"
Nov 02 22:14:52.847401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847395    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55737394-a6d7-4d24-9eb9-11a329fafdd9" containerName="istio-proxy"
Nov 02 22:14:52.847581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847514    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="61816091-d007-4af4-90b6-52cfbe8ea50b" containerName="payload-pre-processing"
Nov 02 22:14:52.847581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847523    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="55737394-a6d7-4d24-9eb9-11a329fafdd9" containerName="istio-proxy"
Nov 02 22:14:52.847581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.847559    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1907a5-a28e-4cff-a97a-9cc9c1d121d4" containerName="istio-proxy"
Nov 02 22:14:52.852559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.852525    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:52.856673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.856649    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7144fe\""
Nov 02 22:14:52.860701 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.860677    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:14:52.870736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.870716    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7144fe-dockercfg-4jtj6\""
Nov 02 22:14:52.888759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.888731    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:14:52.893334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.893311    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:52.904156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.904130    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:14:52.948386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.948349    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:52.948529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.948401    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hm2\" (UniqueName: \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:52.948529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.948516    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:52.948674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:52.948659    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rkp\" (UniqueName: \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.058617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.058500    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rkp\" (UniqueName: \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.084388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.058834    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:53.084388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.058894    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hm2\" (UniqueName: \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:53.084388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.058993    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.084388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.060007    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.084388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.076046    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:53.113103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.112356    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rkp\" (UniqueName: \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\") pod \"payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.143916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.143880    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hm2\" (UniqueName: \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\") pod \"payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:53.186975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.186941    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:53.305287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.304934    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:53.697122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.697077    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:14:53.704110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.704084    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:14:53.706723 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:14:53.706697    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d676b3_6802_4995_b9f0_44e42cbff2d3.slice/crio-1982431da863a69e2ebaf5ca294f8da3ffdef73e35b53ac6f4b9c5136632a5fc WatchSource:0}: Error finding container 1982431da863a69e2ebaf5ca294f8da3ffdef73e35b53ac6f4b9c5136632a5fc: Status 404 returned error can't find the container with id 1982431da863a69e2ebaf5ca294f8da3ffdef73e35b53ac6f4b9c5136632a5fc
Nov 02 22:14:53.723620 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:14:53.723592    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2374b6f_76f5_42ed_a366_3173ffe13c3b.slice/crio-8977c23f738c992ddc87079908d474352985fb337d620ea46da29bfbb4f1864c WatchSource:0}: Error finding container 8977c23f738c992ddc87079908d474352985fb337d620ea46da29bfbb4f1864c: Status 404 returned error can't find the container with id 8977c23f738c992ddc87079908d474352985fb337d620ea46da29bfbb4f1864c
Nov 02 22:14:53.724696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.724596    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:14:53.898557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:53.884589    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:54.188298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:54.184243    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:14:54.735028 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:54.669356    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" event={"ID":"51d676b3-6802-4995-b9f0-44e42cbff2d3","Type":"ContainerStarted","Data":"32b74c28e341ffad413b260d84b9cd269dfa0a9901bc911d645625b3c30bece1"}
Nov 02 22:14:54.735028 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:54.669558    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" event={"ID":"51d676b3-6802-4995-b9f0-44e42cbff2d3","Type":"ContainerStarted","Data":"1982431da863a69e2ebaf5ca294f8da3ffdef73e35b53ac6f4b9c5136632a5fc"}
Nov 02 22:14:54.735028 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:54.706960    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:14:55.313788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:55.306160    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" podStartSLOduration=3.306141112 podStartE2EDuration="3.306141112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:54.874381916 +0000 UTC m=+1032.182472723" watchObservedRunningTime="2026-11-02 22:14:55.306141112 +0000 UTC m=+1032.614231921"
Nov 02 22:14:55.313788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:55.307645    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" event={"ID":"d2374b6f-76f5-42ed-a366-3173ffe13c3b","Type":"ContainerStarted","Data":"47c155319b84e7189950cbf237692d80f35886fb70ba9ec47b027258f8171565"}
Nov 02 22:14:55.313788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:55.307691    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" event={"ID":"d2374b6f-76f5-42ed-a366-3173ffe13c3b","Type":"ContainerStarted","Data":"8977c23f738c992ddc87079908d474352985fb337d620ea46da29bfbb4f1864c"}
Nov 02 22:14:55.313788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:55.307710    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:14:55.313788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:55.311425    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" podStartSLOduration=3.311408781 podStartE2EDuration="3.311408781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:55.123028351 +0000 UTC m=+1032.431119158" watchObservedRunningTime="2026-11-02 22:14:55.311408781 +0000 UTC m=+1032.619499587"
Nov 02 22:14:57.316503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.315148    4929 generic.go:356] "Generic (PLEG): container finished" podID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" containerID="8a52660052e57d497aef683a0054dfe304696905690a5f403bd8e2d495f88918" exitCode=0
Nov 02 22:14:57.316503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.316338    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" event={"ID":"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5","Type":"ContainerDied","Data":"8a52660052e57d497aef683a0054dfe304696905690a5f403bd8e2d495f88918"}
Nov 02 22:14:57.677415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.676367    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:14:57.802522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.788875    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.803702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.803670    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.803901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.803884    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804145    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804277    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804455    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804619    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804732 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804720    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.804938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.804924    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\") pod \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\" (UID: \"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5\") "
Nov 02 22:14:57.837556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.793062    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:57.837556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.825926    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:57.837556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.827370    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:57.839986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.839950    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:57.849811 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.844764    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:57.850300 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:57.850267    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:14:57.868743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.868703    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:57.870459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.870426    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:14:57.898860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.898824    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918204    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918244    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918260    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918274    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918288    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918301    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918316    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.919027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.918329    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:57.921166 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:57.921129    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl" pod "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" (UID: "e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5"). InnerVolumeSpecName "kube-api-access-rmwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:58.021333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.021295    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmwrl\" (UniqueName: \"kubernetes.io/projected/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5-kube-api-access-rmwrl\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:58.377355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.377316    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6" event={"ID":"e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5","Type":"ContainerDied","Data":"726c3c4f5b80318cbd5edf3ebb877b50e53174a40a299900f6f3fa92566b67ef"}
Nov 02 22:14:58.377965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.377944    4929 scope.go:122] "RemoveContainer" containerID="8a52660052e57d497aef683a0054dfe304696905690a5f403bd8e2d495f88918"
Nov 02 22:14:58.378201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.378187    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"
Nov 02 22:14:58.409735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.409702    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:14:58.420888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.420845    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6a72dd63-openshift-default-587d65c99f-zdhl6"]
Nov 02 22:14:58.808673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.808594    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" path="/var/lib/kubelet/pods/e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5/volumes"
Nov 02 22:14:58.885692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:58.885645    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" podUID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" containerName="payload-processing" containerID="cri-o://ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a" gracePeriod=25
Nov 02 22:14:59.184692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.184655    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" podUID="ed49ca6e-3508-403d-bd18-8959890821c6" containerName="payload-pre-processing" containerID="cri-o://b37211911718d1c3370936e5a94bff02b33ff2aa5d350efb186e5f3a0b3452d1" gracePeriod=25
Nov 02 22:14:59.328098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.328070    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:59.388113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.388078    4929 generic.go:356] "Generic (PLEG): container finished" podID="ed49ca6e-3508-403d-bd18-8959890821c6" containerID="b37211911718d1c3370936e5a94bff02b33ff2aa5d350efb186e5f3a0b3452d1" exitCode=0
Nov 02 22:14:59.388489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.388153    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" event={"ID":"ed49ca6e-3508-403d-bd18-8959890821c6","Type":"ContainerDied","Data":"b37211911718d1c3370936e5a94bff02b33ff2aa5d350efb186e5f3a0b3452d1"}
Nov 02 22:14:59.389878 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.389853    4929 generic.go:356] "Generic (PLEG): container finished" podID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" containerID="ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a" exitCode=0
Nov 02 22:14:59.389998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.389923    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"
Nov 02 22:14:59.389998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.389956    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" event={"ID":"49ffd9f9-0e2f-4168-aff9-e2a5a014248b","Type":"ContainerDied","Data":"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"}
Nov 02 22:14:59.390071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.389999    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk" event={"ID":"49ffd9f9-0e2f-4168-aff9-e2a5a014248b","Type":"ContainerDied","Data":"342a27dcb1247abd3a22aa2287a171a10d9e90b271a22c21970a84b0e9aa7d81"}
Nov 02 22:14:59.390071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.390030    4929 scope.go:122] "RemoveContainer" containerID="ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"
Nov 02 22:14:59.402088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.402059    4929 scope.go:122] "RemoveContainer" containerID="ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"
Nov 02 22:14:59.402369 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:14:59.402348    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a\": container with ID starting with ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a not found: ID does not exist" containerID="ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"
Nov 02 22:14:59.402455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.402382    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a"} err="failed to get container status \"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a\": rpc error: code = NotFound desc = could not find container \"ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a\": container with ID starting with ddbf918013a0e80d58413f4b575321a573ac142b8e5c91a1ddafc73cc3b60d2a not found: ID does not exist"
Nov 02 22:14:59.466899 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.466872    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:14:59.483386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.483358    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\" (UniqueName: \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\") pod \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") "
Nov 02 22:14:59.483552 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.483441    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\") pod \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\" (UID: \"49ffd9f9-0e2f-4168-aff9-e2a5a014248b\") "
Nov 02 22:14:59.483827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.483803    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume" pod "49ffd9f9-0e2f-4168-aff9-e2a5a014248b" (UID: "49ffd9f9-0e2f-4168-aff9-e2a5a014248b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:59.484332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.484308    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:59.486653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.486624    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh" pod "49ffd9f9-0e2f-4168-aff9-e2a5a014248b" (UID: "49ffd9f9-0e2f-4168-aff9-e2a5a014248b"). InnerVolumeSpecName "kube-api-access-sldfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:59.585869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.585830    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\" (UniqueName: \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\") pod \"ed49ca6e-3508-403d-bd18-8959890821c6\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") "
Nov 02 22:14:59.586132 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.586113    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\") pod \"ed49ca6e-3508-403d-bd18-8959890821c6\" (UID: \"ed49ca6e-3508-403d-bd18-8959890821c6\") "
Nov 02 22:14:59.586469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.586441    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume" pod "ed49ca6e-3508-403d-bd18-8959890821c6" (UID: "ed49ca6e-3508-403d-bd18-8959890821c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:59.586907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.586883    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed49ca6e-3508-403d-bd18-8959890821c6-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:59.586980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.586913    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldfh\" (UniqueName: \"kubernetes.io/projected/49ffd9f9-0e2f-4168-aff9-e2a5a014248b-kube-api-access-sldfh\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:59.588161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.588137    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn" pod "ed49ca6e-3508-403d-bd18-8959890821c6" (UID: "ed49ca6e-3508-403d-bd18-8959890821c6"). InnerVolumeSpecName "kube-api-access-x4fjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:59.688222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.688188    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4fjn\" (UniqueName: \"kubernetes.io/projected/ed49ca6e-3508-403d-bd18-8959890821c6-kube-api-access-x4fjn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:59.714439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.714354    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:59.719595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.719564    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb16f94c-7c8d4c8b9f-dghnk"]
Nov 02 22:14:59.760135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760104    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:14:59.760850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760828    4929 cpu_manager.go:414] "containerMap: removing container" podUID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" containerName="payload-processing"
Nov 02 22:14:59.760850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760847    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" containerName="payload-processing"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760865    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ed49ca6e-3508-403d-bd18-8959890821c6" containerName="payload-pre-processing"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760871    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed49ca6e-3508-403d-bd18-8959890821c6" containerName="payload-pre-processing"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760885    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" containerName="istio-proxy"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760890    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" containerName="istio-proxy"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.760996    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" containerName="payload-processing"
Nov 02 22:14:59.761006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.761007    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed49ca6e-3508-403d-bd18-8959890821c6" containerName="payload-pre-processing"
Nov 02 22:14:59.761184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.761015    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f4efa2-418a-4f3c-92fe-9b5c6e7a8cb5" containerName="istio-proxy"
Nov 02 22:14:59.764856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.764829    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:14:59.770617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.770572    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7144fe-serving-cert\""
Nov 02 22:14:59.786630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.786595    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:14:59.934779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.927224    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:14:59.934779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.927324    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:14:59.934779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.927561    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:14:59.934779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:14:59.927656    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9w7\" (UniqueName: \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.032619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.029317    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.032619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.029454    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9w7\" (UniqueName: \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.032619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.029578    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.032619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.029664    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.034767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.034732    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.039264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.039214    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.068558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.067567    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9w7\" (UniqueName: \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.068752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.068723    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.088934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.088900    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:00.148848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.148807    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:00.155021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.154990    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.163831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.163805    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\""
Nov 02 22:15:00.170273 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.170241    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:00.181737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.181706    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:00.199045 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.199013    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.207415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.207384    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:00.233012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.232906    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vqt\" (UniqueName: \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.233012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.232957    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.233228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.233055    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnl2\" (UniqueName: \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.233228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.233089    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.334487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.334394    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vqt\" (UniqueName: \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.334487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.334453    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.334756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.334499    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnl2\" (UniqueName: \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.334756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.334524    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.335288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.335268    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.335784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.335765    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.345261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.345224    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vqt\" (UniqueName: \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.345261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.345257    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnl2\" (UniqueName: \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.385942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.385906    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:15:00.386097 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:00.385961    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9088356e_6c6d_4d9b_99b5_91346071aee3.slice/crio-4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4 WatchSource:0}: Error finding container 4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4: Status 404 returned error can't find the container with id 4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4
Nov 02 22:15:00.387579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.387518    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:15:00.414497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.414459    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" event={"ID":"9088356e-6c6d-4d9b-99b5-91346071aee3","Type":"ContainerStarted","Data":"4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4"}
Nov 02 22:15:00.416568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.416516    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l" event={"ID":"ed49ca6e-3508-403d-bd18-8959890821c6","Type":"ContainerDied","Data":"9090cbaacc9d9b31db5de45d93c37b62216c8ee2b5ae6d19fd27c42d22028f7b"}
Nov 02 22:15:00.416737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.416619    4929 scope.go:122] "RemoveContainer" containerID="b37211911718d1c3370936e5a94bff02b33ff2aa5d350efb186e5f3a0b3452d1"
Nov 02 22:15:00.416737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.416651    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"
Nov 02 22:15:00.445828 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.445793    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:15:00.451681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.451651    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb16f94c-57bbfb6455-7fw2l"]
Nov 02 22:15:00.511584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.485864    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:00.553642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.553251    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:00.812309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.812273    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ffd9f9-0e2f-4168-aff9-e2a5a014248b" path="/var/lib/kubelet/pods/49ffd9f9-0e2f-4168-aff9-e2a5a014248b/volumes"
Nov 02 22:15:00.813039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:00.813020    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed49ca6e-3508-403d-bd18-8959890821c6" path="/var/lib/kubelet/pods/ed49ca6e-3508-403d-bd18-8959890821c6/volumes"
Nov 02 22:15:01.090414 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:01.090354    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11d6f2c_d217_4054_ac68_68a4837ba20e.slice/crio-d18b38c66d3646069e62b58d1e8e1bf3d0d8845dfdb99748d608d13b7fa0911c WatchSource:0}: Error finding container d18b38c66d3646069e62b58d1e8e1bf3d0d8845dfdb99748d608d13b7fa0911c: Status 404 returned error can't find the container with id d18b38c66d3646069e62b58d1e8e1bf3d0d8845dfdb99748d608d13b7fa0911c
Nov 02 22:15:01.113563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.112690    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:01.215887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.215840    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:01.427760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.427721    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" event={"ID":"9088356e-6c6d-4d9b-99b5-91346071aee3","Type":"ContainerStarted","Data":"8475892560d0a15caabde785e04d9d403b455a03ff1f1e3ba3715fd2a73578dc"}
Nov 02 22:15:01.428161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.427792    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:01.430026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.430000    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerStarted","Data":"d18b38c66d3646069e62b58d1e8e1bf3d0d8845dfdb99748d608d13b7fa0911c"}
Nov 02 22:15:01.431318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.431292    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerStarted","Data":"62257dffb004504a66a06c524fff57f63019a04d789f90ae984e03607c9c535b"}
Nov 02 22:15:01.448655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:01.448558    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" podStartSLOduration=1.922202007 podStartE2EDuration="2.448528462s" totalImagesPullingTime="526.326455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:01.444783201 +0000 UTC m=+1038.752874009" watchObservedRunningTime="2026-11-02 22:15:01.448528462 +0000 UTC m=+1038.756619266"
Nov 02 22:15:05.529510 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:05.529473    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerStarted","Data":"132a563a6c74a52a83967f702014761db147a5f665da68b2eb22b1338ad1b94a"}
Nov 02 22:15:05.545824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:05.545773    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" podStartSLOduration=1.6424045619999998 podStartE2EDuration="5.545758371s" totalImagesPullingTime="3.903353809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.544061354 +0000 UTC m=+1042.852152160" watchObservedRunningTime="2026-11-02 22:15:05.545758371 +0000 UTC m=+1042.853849176"
Nov 02 22:15:06.303890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.303785    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:15:06.540231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.539554    4929 generic.go:356] "Generic (PLEG): container finished" podID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerID="132a563a6c74a52a83967f702014761db147a5f665da68b2eb22b1338ad1b94a" exitCode=7
Nov 02 22:15:06.540231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.539681    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerDied","Data":"132a563a6c74a52a83967f702014761db147a5f665da68b2eb22b1338ad1b94a"}
Nov 02 22:15:06.540231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.540100    4929 scope.go:122] "RemoveContainer" containerID="132a563a6c74a52a83967f702014761db147a5f665da68b2eb22b1338ad1b94a"
Nov 02 22:15:06.541622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.541593    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerStarted","Data":"011cd7d75c7fb35f216fd064669a4aa761e1475b01338089910b291c09c6caff"}
Nov 02 22:15:06.579866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:06.579803    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" podStartSLOduration=2.199112758 podStartE2EDuration="6.579787052s" totalImagesPullingTime="4.380674294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:06.575057735 +0000 UTC m=+1043.883148543" watchObservedRunningTime="2026-11-02 22:15:06.579787052 +0000 UTC m=+1043.887877856"
Nov 02 22:15:07.320149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.320120    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:15:07.509442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.509372    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:15:07.557707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.557676    4929 generic.go:356] "Generic (PLEG): container finished" podID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerID="908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4" exitCode=7
Nov 02 22:15:07.558121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.557805    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerDied","Data":"908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4"}
Nov 02 22:15:07.558121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.557869    4929 scope.go:122] "RemoveContainer" containerID="132a563a6c74a52a83967f702014761db147a5f665da68b2eb22b1338ad1b94a"
Nov 02 22:15:07.558227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.558169    4929 scope.go:122] "RemoveContainer" containerID="908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4"
Nov 02 22:15:07.561954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.561926    4929 generic.go:356] "Generic (PLEG): container finished" podID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerID="011cd7d75c7fb35f216fd064669a4aa761e1475b01338089910b291c09c6caff" exitCode=60
Nov 02 22:15:07.562193 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.562177    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerDied","Data":"011cd7d75c7fb35f216fd064669a4aa761e1475b01338089910b291c09c6caff"}
Nov 02 22:15:07.562347 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:07.562316    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5_odh-ai-gateway-infra(a11d6f2c-d217-4054-ac68-68a4837ba20e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e"
Nov 02 22:15:07.562699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.562650    4929 scope.go:122] "RemoveContainer" containerID="011cd7d75c7fb35f216fd064669a4aa761e1475b01338089910b291c09c6caff"
Nov 02 22:15:07.586306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.586274    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:15:07.592325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.592300    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.599345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.599313    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-84a94060-openshift-default-dockercfg-kccrv\""
Nov 02 22:15:07.604138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.604110    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734022    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734104    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734139    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734170    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734252    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734299    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.734561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.734453    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.784612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.765527    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5w88\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.784612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.765715    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.871559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.868734    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.871559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.868848    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5w88\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.871559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.868960    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.871559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.869188    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.885557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.883392    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.901556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.897766    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.901556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.897835    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.901556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.897874    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.901556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.898028    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.901556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.898087    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.909555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.907099    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.909555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.908648    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.909555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.909183    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.915422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.910140    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.918883 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.918839    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.971165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.971114    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.972439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.972411    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5w88\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:07.993564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:07.978467    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:08.266128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.263268    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:08.626637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.620164    4929 scope.go:122] "RemoveContainer" containerID="908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4"
Nov 02 22:15:08.626637 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:08.620788    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5_odh-ai-gateway-infra(a11d6f2c-d217-4054-ac68-68a4837ba20e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e"
Nov 02 22:15:08.741555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.721983    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:15:08.741555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.725909    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:08.741555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.726000    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:08.741555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.726065    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:08.791981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:08.791255    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerStarted","Data":"6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"}
Nov 02 22:15:09.802005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.801970    4929 generic.go:356] "Generic (PLEG): container finished" podID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerID="6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0" exitCode=60
Nov 02 22:15:09.802511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.802053    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerDied","Data":"6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"}
Nov 02 22:15:09.802511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.802109    4929 scope.go:122] "RemoveContainer" containerID="011cd7d75c7fb35f216fd064669a4aa761e1475b01338089910b291c09c6caff"
Nov 02 22:15:09.802511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.802318    4929 scope.go:122] "RemoveContainer" containerID="6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"
Nov 02 22:15:09.802699 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:09.802645    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz_odh-ai-gateway-infra(7a484b2c-1c32-4bd7-96e3-3042ce97b0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa"
Nov 02 22:15:09.804169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.804144    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerStarted","Data":"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"}
Nov 02 22:15:09.804272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.804190    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerStarted","Data":"4d92b4448d465ecc10646f2b6d228f0661a8afb9255d5e46e449731adb993fde"}
Nov 02 22:15:09.836384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:09.836335    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podStartSLOduration=2.836319478 podStartE2EDuration="2.836319478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:09.834838942 +0000 UTC m=+1047.142929747" watchObservedRunningTime="2026-11-02 22:15:09.836319478 +0000 UTC m=+1047.144410282"
Nov 02 22:15:10.264903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:10.264869    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:10.270266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:10.270239    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:10.813921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:10.813894    4929 scope.go:122] "RemoveContainer" containerID="6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"
Nov 02 22:15:10.814391 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:10.814064    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:10.814391 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:10.814302    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz_odh-ai-gateway-infra(7a484b2c-1c32-4bd7-96e3-3042ce97b0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa"
Nov 02 22:15:10.815039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:10.815019    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:15:12.342468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.342430    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:15:12.354866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.354832    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.367819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.365826    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-334860-openshift-default-dockercfg-tlf4t\""
Nov 02 22:15:12.369868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.369836    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:15:12.477064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477028    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477180    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477229    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477491 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477471    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477514    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477585    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsknw\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477705    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477762    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.477844 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.477828    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.579706    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.579787    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.579885    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.579929    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.580293    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.580344    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.580386    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsknw\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.580504    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.580599    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.581557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.581452    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.623558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.617467    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.623558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.617780    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.623558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.618243    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.623558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.618340    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.674085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.669791    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.674085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.670292    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.674085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.670857    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.677558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.677271    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsknw\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\") pod \"e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") " pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:12.692221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:12.690878    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:13.371548 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.371498    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:15:13.377608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.377579    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.387268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.387239    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-84a94060\""
Nov 02 22:15:13.392031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.392005    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:15:13.415040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.415001    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:15:13.416514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.416471    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-84a94060-dockercfg-2hfpf\""
Nov 02 22:15:13.428402 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:13.424414    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda861bf8d_832d_4f34_8ae9_643e56c1f632.slice/crio-ac9f84aa43f31cc3e344df78f8ec22b1955c419515bec8a652a147ee42a8f90e WatchSource:0}: Error finding container ac9f84aa43f31cc3e344df78f8ec22b1955c419515bec8a652a147ee42a8f90e: Status 404 returned error can't find the container with id ac9f84aa43f31cc3e344df78f8ec22b1955c419515bec8a652a147ee42a8f90e
Nov 02 22:15:13.428870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.428832    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:15:13.428976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.428960    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.435344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.434614    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:13.435344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.434707    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:13.436545 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.435612    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:13.436545 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.435761    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:15:13.665616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.664871    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.665616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.665149    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws98w\" (UniqueName: \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.665616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.665257    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4dz\" (UniqueName: \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.665616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.665444    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.784556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.777600    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws98w\" (UniqueName: \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.784556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.777731    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4dz\" (UniqueName: \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.784556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.777900    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.784556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.777992    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.784556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.779074    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.820559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.810368    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.850855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.850457    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4dz\" (UniqueName: \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:13.875410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.875379    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws98w\" (UniqueName: \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:13.923570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:13.923098    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerStarted","Data":"ac9f84aa43f31cc3e344df78f8ec22b1955c419515bec8a652a147ee42a8f90e"}
Nov 02 22:15:14.018081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.017992    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:14.082480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.071154    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:14.451384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.451352    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:15:14.452339 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:14.452312    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fc6993_bf61_499d_bf3e_4654cdda1cee.slice/crio-5aa4706894ac067bc4adc93304dfcc4cb6fb82ea1ce42d1119636da48fa49a94 WatchSource:0}: Error finding container 5aa4706894ac067bc4adc93304dfcc4cb6fb82ea1ce42d1119636da48fa49a94: Status 404 returned error can't find the container with id 5aa4706894ac067bc4adc93304dfcc4cb6fb82ea1ce42d1119636da48fa49a94
Nov 02 22:15:14.481891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.481864    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:15:14.489087 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:14.489056    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6efa77_19ef_46cb_b642_0f67b0e294dd.slice/crio-4722eba8ad1a07463fce66f899bc0c546b1aa9b3438583deff425a0d60e0b57a WatchSource:0}: Error finding container 4722eba8ad1a07463fce66f899bc0c546b1aa9b3438583deff425a0d60e0b57a: Status 404 returned error can't find the container with id 4722eba8ad1a07463fce66f899bc0c546b1aa9b3438583deff425a0d60e0b57a
Nov 02 22:15:14.932516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.932478    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" event={"ID":"3d6efa77-19ef-46cb-b642-0f67b0e294dd","Type":"ContainerStarted","Data":"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"}
Nov 02 22:15:14.932754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.932529    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" event={"ID":"3d6efa77-19ef-46cb-b642-0f67b0e294dd","Type":"ContainerStarted","Data":"4722eba8ad1a07463fce66f899bc0c546b1aa9b3438583deff425a0d60e0b57a"}
Nov 02 22:15:14.932754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.932733    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:14.934325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.934302    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" event={"ID":"74fc6993-bf61-499d-bf3e-4654cdda1cee","Type":"ContainerStarted","Data":"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"}
Nov 02 22:15:14.934469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.934339    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" event={"ID":"74fc6993-bf61-499d-bf3e-4654cdda1cee","Type":"ContainerStarted","Data":"5aa4706894ac067bc4adc93304dfcc4cb6fb82ea1ce42d1119636da48fa49a94"}
Nov 02 22:15:14.934469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.934381    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:14.936027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.936007    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerStarted","Data":"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"}
Nov 02 22:15:14.949651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.949604    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" podStartSLOduration=1.949591136 podStartE2EDuration="1.949591136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:14.947500669 +0000 UTC m=+1052.255591477" watchObservedRunningTime="2026-11-02 22:15:14.949591136 +0000 UTC m=+1052.257681940"
Nov 02 22:15:14.973903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.973853    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podStartSLOduration=2.9738409260000003 podStartE2EDuration="2.973840926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:14.969199281 +0000 UTC m=+1052.277290087" watchObservedRunningTime="2026-11-02 22:15:14.973840926 +0000 UTC m=+1052.281931791"
Nov 02 22:15:14.985461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:14.985407    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" podStartSLOduration=1.9853899130000001 podStartE2EDuration="1.985389913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:14.983924127 +0000 UTC m=+1052.292014937" watchObservedRunningTime="2026-11-02 22:15:14.985389913 +0000 UTC m=+1052.293480722"
Nov 02 22:15:15.696600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:15.696562    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:15.702499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:15.702472    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:15.943848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:15.943811    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:15.944827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:15.944803    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:15:18.394220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.394184    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:15:18.399082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.399061    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.404267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.404245    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-334860-dockercfg-mnwnb\""
Nov 02 22:15:18.404380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.404306    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-334860\""
Nov 02 22:15:18.409353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.409070    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:15:18.429644 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.429604    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:15:18.437955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.437923    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:15:18.438310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.438288    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.482638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.482605    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhq55\" (UniqueName: \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.482638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.482642    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.590560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.589661    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhq55\" (UniqueName: \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.590560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.589725    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.590560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.589826    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9l6\" (UniqueName: \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.590560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.589880    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.604556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.591505    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.652375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.652256    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhq55\" (UniqueName: \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\") pod \"payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.703559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.697847    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9l6\" (UniqueName: \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.703559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.697920    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.703559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.698882    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.735565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.732111    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:18.735565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.733693    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9l6\" (UniqueName: \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\") pod \"payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:18.787558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:18.774384    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:19.211449 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.211374    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:15:19.225014 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:19.224986    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a28435b_cae8_4687_b858_d3f6796d44c7.slice/crio-d37bf4b56d6a02276c41b05f3b321f1ecb1730c252cec44ec7cdcb6bc72a987c WatchSource:0}: Error finding container d37bf4b56d6a02276c41b05f3b321f1ecb1730c252cec44ec7cdcb6bc72a987c: Status 404 returned error can't find the container with id d37bf4b56d6a02276c41b05f3b321f1ecb1730c252cec44ec7cdcb6bc72a987c
Nov 02 22:15:19.266577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.241404    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:15:19.982255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.982214    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" event={"ID":"e99d0205-b9ce-41f2-857c-2610e6b62868","Type":"ContainerStarted","Data":"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"}
Nov 02 22:15:19.982735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.982279    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" event={"ID":"e99d0205-b9ce-41f2-857c-2610e6b62868","Type":"ContainerStarted","Data":"81aae65505f0b41eeb318d7723c15147acd8f20b521c0aebc0d05fdc3265aace"}
Nov 02 22:15:19.982735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.982308    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:19.984022 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.983999    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" event={"ID":"5a28435b-cae8-4687-b858-d3f6796d44c7","Type":"ContainerStarted","Data":"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"}
Nov 02 22:15:19.984141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.984045    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" event={"ID":"5a28435b-cae8-4687-b858-d3f6796d44c7","Type":"ContainerStarted","Data":"d37bf4b56d6a02276c41b05f3b321f1ecb1730c252cec44ec7cdcb6bc72a987c"}
Nov 02 22:15:19.984141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:19.984086    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:20.024972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:20.024922    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" podStartSLOduration=2.024907709 podStartE2EDuration="2.024907709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:20.000472531 +0000 UTC m=+1057.308563356" watchObservedRunningTime="2026-11-02 22:15:20.024907709 +0000 UTC m=+1057.332998513"
Nov 02 22:15:20.025991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:20.025924    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" podStartSLOduration=2.025908695 podStartE2EDuration="2.025908695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:20.021477672 +0000 UTC m=+1057.329568476" watchObservedRunningTime="2026-11-02 22:15:20.025908695 +0000 UTC m=+1057.333999506"
Nov 02 22:15:20.807619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:20.807585    4929 scope.go:122] "RemoveContainer" containerID="908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4"
Nov 02 22:15:22.007037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:22.007002    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerStarted","Data":"f7486c57ec75596d9cf3f6afc2e842f46a3afe00b688457802f842babf3f5fe6"}
Nov 02 22:15:22.883991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:22.883959    4929 scope.go:122] "RemoveContainer" containerID="6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"
Nov 02 22:15:23.018017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:23.017978    4929 generic.go:356] "Generic (PLEG): container finished" podID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerID="f7486c57ec75596d9cf3f6afc2e842f46a3afe00b688457802f842babf3f5fe6" exitCode=60
Nov 02 22:15:23.018502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:23.018042    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerDied","Data":"f7486c57ec75596d9cf3f6afc2e842f46a3afe00b688457802f842babf3f5fe6"}
Nov 02 22:15:23.018502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:23.018104    4929 scope.go:122] "RemoveContainer" containerID="908c08a931527500ada6e5c8903fbf4c4559a15abfc911eb8a3826f16cb7eee4"
Nov 02 22:15:23.018695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:23.018597    4929 scope.go:122] "RemoveContainer" containerID="f7486c57ec75596d9cf3f6afc2e842f46a3afe00b688457802f842babf3f5fe6"
Nov 02 22:15:23.019170 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:23.019052    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5_odh-ai-gateway-infra(a11d6f2c-d217-4054-ac68-68a4837ba20e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e"
Nov 02 22:15:23.307641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:23.289302    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:24.039553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.039395    4929 generic.go:356] "Generic (PLEG): container finished" podID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerID="a9ee1edb12793e4f1a04fbc399794ca1675aa8874b57c64165e0668b93f31146" exitCode=60
Nov 02 22:15:24.039553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.039454    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerDied","Data":"a9ee1edb12793e4f1a04fbc399794ca1675aa8874b57c64165e0668b93f31146"}
Nov 02 22:15:24.066635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.039607    4929 scope.go:122] "RemoveContainer" containerID="6b9a0bc9f68aebfa4ea66b3180ea07acb0e554df6b53ceb9967ccc949f9443f0"
Nov 02 22:15:24.066635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.040625    4929 scope.go:122] "RemoveContainer" containerID="a9ee1edb12793e4f1a04fbc399794ca1675aa8874b57c64165e0668b93f31146"
Nov 02 22:15:24.066635 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:24.041225    4929 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz_odh-ai-gateway-infra(7a484b2c-1c32-4bd7-96e3-3042ce97b0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa"
Nov 02 22:15:24.744677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.744644    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:24.933672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.933627    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\" (UniqueName: \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\") pod \"a11d6f2c-d217-4054-ac68-68a4837ba20e\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") "
Nov 02 22:15:24.934220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.934199    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\") pod \"a11d6f2c-d217-4054-ac68-68a4837ba20e\" (UID: \"a11d6f2c-d217-4054-ac68-68a4837ba20e\") "
Nov 02 22:15:24.952582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.951786    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca" pod "a11d6f2c-d217-4054-ac68-68a4837ba20e" (UID: "a11d6f2c-d217-4054-ac68-68a4837ba20e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:24.992879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:24.982992    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt" pod "a11d6f2c-d217-4054-ac68-68a4837ba20e" (UID: "a11d6f2c-d217-4054-ac68-68a4837ba20e"). InnerVolumeSpecName "kube-api-access-z5vqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:25.081813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.081735    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11d6f2c-d217-4054-ac68-68a4837ba20e-maas-api-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:25.081813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.081772    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vqt\" (UniqueName: \"kubernetes.io/projected/a11d6f2c-d217-4054-ac68-68a4837ba20e-kube-api-access-z5vqt\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:25.133582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.133225    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:25.216616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.211410    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5" event={"ID":"a11d6f2c-d217-4054-ac68-68a4837ba20e","Type":"ContainerDied","Data":"d18b38c66d3646069e62b58d1e8e1bf3d0d8845dfdb99748d608d13b7fa0911c"}
Nov 02 22:15:25.216616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.211495    4929 scope.go:122] "RemoveContainer" containerID="f7486c57ec75596d9cf3f6afc2e842f46a3afe00b688457802f842babf3f5fe6"
Nov 02 22:15:25.249625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.230201    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"
Nov 02 22:15:25.516799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.516716    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:25.614547 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:25.613947    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7144fe-29894295-t6zt5"]
Nov 02 22:15:26.108486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.107606    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.151603    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.152976    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153003    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153050    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153059    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153079    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153087    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153322    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153342    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.162563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.153354    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" containerName="cleanup"
Nov 02 22:15:26.236156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.217745    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:15:26.429150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.309616    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:26.429150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.358824    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:15:26.429150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.358922    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:15:26.429150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.363666    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:26.485333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.466807    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-334860-serving-cert\""
Nov 02 22:15:26.520213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.519668    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:15:26.675336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:26.538218    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.049588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.020800    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.140896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.129103    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:15:27.229720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.229672    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:27.291557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.290640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.317615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.292707    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.318236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.318185    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:27.365779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.346920    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.368313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.368275    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.368510 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.368466    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.380558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.368913    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6r5b\" (UniqueName: \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.381090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.369104    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:27.434263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.434085    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.510570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.509407    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:15:27.546971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.546926    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:27.672559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.658984    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:27.744794 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:27.744743    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:15:27.771558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.767941    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.803771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.803730    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6r5b\" (UniqueName: \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.804053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.804028    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.804482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.804458    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.805672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.805648    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.840558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.840246    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.864559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.850140    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.864559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.856136    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:27.877558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.867599    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11d6f2c-d217-4054-ac68-68a4837ba20e" path="/var/lib/kubelet/pods/a11d6f2c-d217-4054-ac68-68a4837ba20e/volumes"
Nov 02 22:15:27.877558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.868273    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:15:27.877558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.868331    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:15:27.877558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.868364    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:15:27.877558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.868400    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:15:27.900944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.900908    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6r5b\" (UniqueName: \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\") pod \"maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:27.911558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.907735    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\") pod \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") "
Nov 02 22:15:28.012615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.011074    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\" (UniqueName: \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\") pod \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\" (UID: \"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa\") "
Nov 02 22:15:28.025597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.929288    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:28.088589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.994687    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:15:28.370578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.026669    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:27.941103    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca" pod "7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" (UID: "7a484b2c-1c32-4bd7-96e3-3042ce97b0fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.072875    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.375723    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.358020    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.359884    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.368677    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.386187    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.386271    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.388273    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.388701    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:15:28.461706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.447257    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-maas-api-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:28.594076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.574239    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:15:28.729919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.729886    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2" pod "7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" (UID: "7a484b2c-1c32-4bd7-96e3-3042ce97b0fa"). InnerVolumeSpecName "kube-api-access-ctnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:28.812668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.810388    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 22:15:28.825653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.825088    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:15:28.942590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.925559    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"
Nov 02 22:15:28.979682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:28.926373    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctnl2\" (UniqueName: \"kubernetes.io/projected/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa-kube-api-access-ctnl2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:29.004711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.004672    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:15:29.006723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.006697    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:15:29.011806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.011789    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:15:29.011964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.011947    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz" event={"ID":"7a484b2c-1c32-4bd7-96e3-3042ce97b0fa","Type":"ContainerDied","Data":"62257dffb004504a66a06c524fff57f63019a04d789f90ae984e03607c9c535b"}
Nov 02 22:15:29.015574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.015552    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:15:29.015788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.015774    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:15:29.016155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.015716    4929 scope.go:122] "RemoveContainer" containerID="a9ee1edb12793e4f1a04fbc399794ca1675aa8874b57c64165e0668b93f31146"
Nov 02 22:15:29.314997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.314879    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:29.320339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.320302    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c302b6-29894295-9xbzz"]
Nov 02 22:15:29.463894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:29.463866    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:15:29.467810 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:29.467774    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757a2431_fba1_4c0f_ad05_e563ac8c8d47.slice/crio-196a26cacd8a3fad0da27629fe9e703201ec0e4899378b55bfb7e98a55ea0de8 WatchSource:0}: Error finding container 196a26cacd8a3fad0da27629fe9e703201ec0e4899378b55bfb7e98a55ea0de8: Status 404 returned error can't find the container with id 196a26cacd8a3fad0da27629fe9e703201ec0e4899378b55bfb7e98a55ea0de8
Nov 02 22:15:30.179064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:30.179027    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" event={"ID":"757a2431-fba1-4c0f-ad05-e563ac8c8d47","Type":"ContainerStarted","Data":"196a26cacd8a3fad0da27629fe9e703201ec0e4899378b55bfb7e98a55ea0de8"}
Nov 02 22:15:30.807625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:30.807593    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" path="/var/lib/kubelet/pods/7a484b2c-1c32-4bd7-96e3-3042ce97b0fa/volumes"
Nov 02 22:15:30.999374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:30.999292    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:15:30.999374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:30.999375    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:15:31.188149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:31.188110    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" event={"ID":"757a2431-fba1-4c0f-ad05-e563ac8c8d47","Type":"ContainerStarted","Data":"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"}
Nov 02 22:15:31.188343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:31.188181    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:31.209272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:31.209216    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" podStartSLOduration=5.592591849 podStartE2EDuration="6.209201854s" totalImagesPullingTime="616.610005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:31.205875302 +0000 UTC m=+1068.513966111" watchObservedRunningTime="2026-11-02 22:15:31.209201854 +0000 UTC m=+1068.517292659"
Nov 02 22:15:33.681985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.681951    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:15:33.682941 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.682919    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683093    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683119    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683124    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683150    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683156    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683334    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683344    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.683380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.683353    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a484b2c-1c32-4bd7-96e3-3042ce97b0fa" containerName="cleanup"
Nov 02 22:15:33.704043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.703892    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:15:33.707993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.707520    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.719252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.719226    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84a94060\"/\"openshift-service-ca.crt\""
Nov 02 22:15:33.719360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.719327    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84a94060\"/\"e2e-lifecycle-model-84a94060-kserve-self-signed-certs\""
Nov 02 22:15:33.719360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.719357    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84a94060\"/\"kube-root-ca.crt\""
Nov 02 22:15:33.719454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.719406    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84a94060\"/\"default-dockercfg-fnqj2\""
Nov 02 22:15:33.724042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724014    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.724156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724065    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.724526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724295    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.724526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724348    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.724526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724485    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.724743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.724587    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptxz\" (UniqueName: \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.826382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.826332    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptxz\" (UniqueName: \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.826620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.826518    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.826749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.826714    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.826909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.826890    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.826985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.826931    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.827044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.827003    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.827152 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.827110    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.827241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.827214    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.827336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.827312    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.829348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.829321    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.829587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.829569    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:33.834814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:33.834791    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptxz\" (UniqueName: \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\") pod \"e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") " pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:34.040020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:34.037839    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:34.664404 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:34.664367    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06984497_c159_4407_8bdd_10db029492ed.slice/crio-95805c86c292cf1108aa824c2fe780278e406806c7e48cd5fe490020174f171c WatchSource:0}: Error finding container 95805c86c292cf1108aa824c2fe780278e406806c7e48cd5fe490020174f171c: Status 404 returned error can't find the container with id 95805c86c292cf1108aa824c2fe780278e406806c7e48cd5fe490020174f171c
Nov 02 22:15:34.664724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:34.664697    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:15:35.238431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:35.238383    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" event={"ID":"06984497-c159-4407-8bdd-10db029492ed","Type":"ContainerStarted","Data":"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"}
Nov 02 22:15:35.238972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:35.238447    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" event={"ID":"06984497-c159-4407-8bdd-10db029492ed","Type":"ContainerStarted","Data":"95805c86c292cf1108aa824c2fe780278e406806c7e48cd5fe490020174f171c"}
Nov 02 22:15:35.259590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:35.259515    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podStartSLOduration=2.259496533 podStartE2EDuration="2.259496533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:35.255388317 +0000 UTC m=+1072.563479122" watchObservedRunningTime="2026-11-02 22:15:35.259496533 +0000 UTC m=+1072.567587339"
Nov 02 22:15:35.909791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:35.906689    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": read tcp 10.133.0.2:48880->10.133.0.97:15021: read: connection reset by peer" start-of-body=
Nov 02 22:15:35.909791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:35.906745    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": read tcp 10.133.0.2:48880->10.133.0.97:15021: read: connection reset by peer"
Nov 02 22:15:36.038779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.038740    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:36.248602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.248509    4929 generic.go:356] "Generic (PLEG): container finished" podID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerID="701d7efc37732cd8e4c19a3f4c199ffa945de7931a3fda379cc83c9212c68ebe" exitCode=0
Nov 02 22:15:36.249014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.248595    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerDied","Data":"701d7efc37732cd8e4c19a3f4c199ffa945de7931a3fda379cc83c9212c68ebe"}
Nov 02 22:15:36.249426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.249403    4929 scope.go:122] "RemoveContainer" containerID="701d7efc37732cd8e4c19a3f4c199ffa945de7931a3fda379cc83c9212c68ebe"
Nov 02 22:15:36.251042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.251012    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:36.251109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.251085    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:36.251187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:36.251173    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:15:37.209755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:37.209725    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:15:37.278308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:37.278255    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerStarted","Data":"c3aff3411aaa441820bc4fd3b9e0e7ead94e2668a322d2e023b34592effc2775"}
Nov 02 22:15:38.036561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:38.036214    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:15:38.036561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:38.036370    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 22:15:38.036561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:38.036468    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 22:15:39.032329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:39.032297    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 22:15:39.032799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:39.032349    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 22:15:40.032631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:40.032598    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 22:15:40.033056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:40.032649    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 22:15:41.042561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:41.042083    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 22:15:41.042561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:41.042147    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 22:15:42.032875 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:42.032837    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 22:15:42.033086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:42.032896    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 22:15:43.036650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:43.036617    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:15:43.037159 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:43.037137    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:15:43.038080 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:43.038056    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:15:44.051558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:44.048460    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:44.109854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:44.108205    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:44.374671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:44.374641    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:15:48.516032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.516001    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6965-0a7a23"]
Nov 02 22:15:48.525726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.525701    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:48.526916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.526879    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-0a7a23"]
Nov 02 22:15:48.529153 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.529120    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:15:48.529265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.529162    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:15:48.530304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.530283    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8p5jl\""
Nov 02 22:15:48.568599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.568560    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25hj\" (UniqueName: \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\") pod \"test-curl-6965-0a7a23\" (UID: \"5560b3b4-b8f2-4919-b945-db84dd5bf84e\") " pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:48.670959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.670926    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25hj\" (UniqueName: \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\") pod \"test-curl-6965-0a7a23\" (UID: \"5560b3b4-b8f2-4919-b945-db84dd5bf84e\") " pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:48.678423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.678396    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25hj\" (UniqueName: \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\") pod \"test-curl-6965-0a7a23\" (UID: \"5560b3b4-b8f2-4919-b945-db84dd5bf84e\") " pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:48.850604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:48.841848    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:49.159523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:49.159497    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-0a7a23"]
Nov 02 22:15:49.159523 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:49.159508    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5560b3b4_b8f2_4919_b945_db84dd5bf84e.slice/crio-18d4bbfeb8d324b2777b3aa8f8ba141250fbdcb81af591879553dc2a10b36632 WatchSource:0}: Error finding container 18d4bbfeb8d324b2777b3aa8f8ba141250fbdcb81af591879553dc2a10b36632: Status 404 returned error can't find the container with id 18d4bbfeb8d324b2777b3aa8f8ba141250fbdcb81af591879553dc2a10b36632
Nov 02 22:15:49.427609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:49.424458    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-0a7a23" event={"ID":"5560b3b4-b8f2-4919-b945-db84dd5bf84e","Type":"ContainerStarted","Data":"18d4bbfeb8d324b2777b3aa8f8ba141250fbdcb81af591879553dc2a10b36632"}
Nov 02 22:15:50.962622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:50.962579    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:15:53.514575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:53.514516    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-0a7a23" event={"ID":"5560b3b4-b8f2-4919-b945-db84dd5bf84e","Type":"ContainerStarted","Data":"04f728fda011a455eb9dc63d5169d39f9eec436c64e767ae5d2f054ad8faa6ad"}
Nov 02 22:15:53.529481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:53.529427    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6965-0a7a23" podStartSLOduration=1.915012852 podStartE2EDuration="5.529412467s" totalImagesPullingTime="3.614399615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:53.527750724 +0000 UTC m=+1090.835841531" watchObservedRunningTime="2026-11-02 22:15:53.529412467 +0000 UTC m=+1090.837503271"
Nov 02 22:15:54.490572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.490381    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6965-0a7a23"]
Nov 02 22:15:54.495318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.495289    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6965-0a7a23"]
Nov 02 22:15:54.734653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.734610    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6965-a8b01d"]
Nov 02 22:15:54.776653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.736380    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" containerName="test-curl-6965-0a7a23"
Nov 02 22:15:54.776653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.772315    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" containerName="test-curl-6965-0a7a23"
Nov 02 22:15:54.776653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.772597    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" containerName="test-curl-6965-0a7a23"
Nov 02 22:15:54.899871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.899836    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:54.915992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:54.915964    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-a8b01d"]
Nov 02 22:15:55.032787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.032688    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgck8\" (UniqueName: \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\") pod \"test-curl-6965-a8b01d\" (UID: \"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9\") " pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:55.142559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.135052    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgck8\" (UniqueName: \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\") pod \"test-curl-6965-a8b01d\" (UID: \"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9\") " pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:55.189711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.189512    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgck8\" (UniqueName: \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\") pod \"test-curl-6965-a8b01d\" (UID: \"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9\") " pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:55.218219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.218186    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:55.603638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.602947    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6965-0a7a23" podUID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" containerName="test-curl-6965-0a7a23" containerID="cri-o://04f728fda011a455eb9dc63d5169d39f9eec436c64e767ae5d2f054ad8faa6ad" gracePeriod=2
Nov 02 22:15:55.795426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:55.795267    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-a8b01d"]
Nov 02 22:15:55.803399 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:55.803366    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33a9a23e_22e5_4122_b5b6_e6f2dfa134b9.slice/crio-8dcba3260c2d8a792e61aa3f4dbed7bf58af5e18c97502d085147fc55c5c5d97 WatchSource:0}: Error finding container 8dcba3260c2d8a792e61aa3f4dbed7bf58af5e18c97502d085147fc55c5c5d97: Status 404 returned error can't find the container with id 8dcba3260c2d8a792e61aa3f4dbed7bf58af5e18c97502d085147fc55c5c5d97
Nov 02 22:15:56.650557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:56.650434    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-a8b01d" event={"ID":"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9","Type":"ContainerStarted","Data":"c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd"}
Nov 02 22:15:56.650557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:56.650491    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-a8b01d" event={"ID":"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9","Type":"ContainerStarted","Data":"8dcba3260c2d8a792e61aa3f4dbed7bf58af5e18c97502d085147fc55c5c5d97"}
Nov 02 22:15:57.514813 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:57.514768    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:15:57.538683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.538134    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6965-a8b01d" podStartSLOduration=3.538112725 podStartE2EDuration="3.538112725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:56.702731383 +0000 UTC m=+1094.010822191" watchObservedRunningTime="2026-11-02 22:15:57.538112725 +0000 UTC m=+1094.846203582"
Nov 02 22:15:57.578006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.574870    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6965-a8b01d"]
Nov 02 22:15:57.586699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.586650    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6965-a8b01d"]
Nov 02 22:15:57.714164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.714130    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6965-128625"]
Nov 02 22:15:57.717557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.716499    4929 cpu_manager.go:414] "containerMap: removing container" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" containerName="test-curl-6965-a8b01d"
Nov 02 22:15:57.717557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.716524    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" containerName="test-curl-6965-a8b01d"
Nov 02 22:15:57.717557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.716964    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" containerName="test-curl-6965-a8b01d"
Nov 02 22:15:57.746557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.737944    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-128625"
Nov 02 22:15:57.790659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.754402    4929 generic.go:356] "Generic (PLEG): container finished" podID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" containerID="04f728fda011a455eb9dc63d5169d39f9eec436c64e767ae5d2f054ad8faa6ad" exitCode=137
Nov 02 22:15:57.855029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.854992    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnvl\" (UniqueName: \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\") pod \"test-curl-6965-128625\" (UID: \"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a\") " pod="opendatahub/test-curl-6965-128625"
Nov 02 22:15:57.887838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:57.886590    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-128625"]
Nov 02 22:15:58.094669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.094634    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnvl\" (UniqueName: \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\") pod \"test-curl-6965-128625\" (UID: \"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a\") " pod="opendatahub/test-curl-6965-128625"
Nov 02 22:15:58.116471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.116445    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnvl\" (UniqueName: \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\") pod \"test-curl-6965-128625\" (UID: \"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a\") " pod="opendatahub/test-curl-6965-128625"
Nov 02 22:15:58.129043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.129001    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-128625"
Nov 02 22:15:58.313779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.313756    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:58.416456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.416396    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\" (UniqueName: \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\") pod \"5560b3b4-b8f2-4919-b945-db84dd5bf84e\" (UID: \"5560b3b4-b8f2-4919-b945-db84dd5bf84e\") "
Nov 02 22:15:58.599556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.599137    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj" pod "5560b3b4-b8f2-4919-b945-db84dd5bf84e" (UID: "5560b3b4-b8f2-4919-b945-db84dd5bf84e"). InnerVolumeSpecName "kube-api-access-q25hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:58.693191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.693107    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q25hj\" (UniqueName: \"kubernetes.io/projected/5560b3b4-b8f2-4919-b945-db84dd5bf84e-kube-api-access-q25hj\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:58.699765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.699735    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-128625"]
Nov 02 22:15:58.700477 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:15:58.700446    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3ccb7e2_33bd_4c1f_bfa9_99f2d467861a.slice/crio-9556916c7803f10f21bf44c534df98f5a3b121945486ab409c2da528f2c409a0 WatchSource:0}: Error finding container 9556916c7803f10f21bf44c534df98f5a3b121945486ab409c2da528f2c409a0: Status 404 returned error can't find the container with id 9556916c7803f10f21bf44c534df98f5a3b121945486ab409c2da528f2c409a0
Nov 02 22:15:58.805089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.805031    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6965-a8b01d" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" containerName="test-curl-6965-a8b01d" containerID="cri-o://c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd" gracePeriod=2
Nov 02 22:15:58.805282 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.805103    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-0a7a23"
Nov 02 22:15:58.807523 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:58.807469    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-a8b01d\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:58.809737 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:15:58.809689    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-a8b01d\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:15:58.812755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.812721    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5560b3b4-b8f2-4919-b945-db84dd5bf84e" path="/var/lib/kubelet/pods/5560b3b4-b8f2-4919-b945-db84dd5bf84e/volumes"
Nov 02 22:15:58.813722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.813695    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-128625" event={"ID":"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a","Type":"ContainerStarted","Data":"9556916c7803f10f21bf44c534df98f5a3b121945486ab409c2da528f2c409a0"}
Nov 02 22:15:58.813827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:58.813751    4929 scope.go:122] "RemoveContainer" containerID="04f728fda011a455eb9dc63d5169d39f9eec436c64e767ae5d2f054ad8faa6ad"
Nov 02 22:15:59.815350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:59.815305    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-128625" event={"ID":"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a","Type":"ContainerStarted","Data":"4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a"}
Nov 02 22:15:59.832285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:15:59.832218    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6965-128625" podStartSLOduration=2.832202926 podStartE2EDuration="2.832202926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:59.829971676 +0000 UTC m=+1097.138062482" watchObservedRunningTime="2026-11-02 22:15:59.832202926 +0000 UTC m=+1097.140293730"
Nov 02 22:16:00.639315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.639085    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6965-128625"]
Nov 02 22:16:00.643735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.643652    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6965-128625"]
Nov 02 22:16:00.784634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.784598    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6965-bee2c1"]
Nov 02 22:16:00.785418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.785395    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" containerName="test-curl-6965-128625"
Nov 02 22:16:00.785418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.785414    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" containerName="test-curl-6965-128625"
Nov 02 22:16:00.785605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.785584    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" containerName="test-curl-6965-128625"
Nov 02 22:16:00.801055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.790369    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:00.820082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.820019    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-bee2c1"]
Nov 02 22:16:00.864387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.864338    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvjq\" (UniqueName: \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\") pod \"test-curl-6965-bee2c1\" (UID: \"2d530518-5a6c-4fb5-a1d8-3506a8710322\") " pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:00.966415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.966371    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvjq\" (UniqueName: \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\") pod \"test-curl-6965-bee2c1\" (UID: \"2d530518-5a6c-4fb5-a1d8-3506a8710322\") " pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:00.979345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:00.978321    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvjq\" (UniqueName: \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\") pod \"test-curl-6965-bee2c1\" (UID: \"2d530518-5a6c-4fb5-a1d8-3506a8710322\") " pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:01.082708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.082677    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:16:01.113964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.113915    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:01.172297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.172238    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\" (UniqueName: \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\") pod \"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9\" (UID: \"33a9a23e-22e5-4122-b5b6-e6f2dfa134b9\") "
Nov 02 22:16:01.254614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.253753    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8" pod "33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" (UID: "33a9a23e-22e5-4122-b5b6-e6f2dfa134b9"). InnerVolumeSpecName "kube-api-access-jgck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:01.291455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.291413    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgck8\" (UniqueName: \"kubernetes.io/projected/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9-kube-api-access-jgck8\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:01.526631 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:16:01.526597    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d530518_5a6c_4fb5_a1d8_3506a8710322.slice/crio-ca12800cd9899766b8d128941bc4e1ffff3bf0c2971a8514fc3d12cd7d0d68ab WatchSource:0}: Error finding container ca12800cd9899766b8d128941bc4e1ffff3bf0c2971a8514fc3d12cd7d0d68ab: Status 404 returned error can't find the container with id ca12800cd9899766b8d128941bc4e1ffff3bf0c2971a8514fc3d12cd7d0d68ab
Nov 02 22:16:01.528963 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.528770    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6965-bee2c1"]
Nov 02 22:16:01.875557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.873249    4929 generic.go:356] "Generic (PLEG): container finished" podID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" containerID="c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd" exitCode=137
Nov 02 22:16:01.875557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.873466    4929 scope.go:122] "RemoveContainer" containerID="c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd"
Nov 02 22:16:01.876245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.875590    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:16:01.899081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.893169    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6965-128625" podUID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" containerName="test-curl-6965-128625" containerID="cri-o://4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a" gracePeriod=2
Nov 02 22:16:01.899081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.893406    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-bee2c1" event={"ID":"2d530518-5a6c-4fb5-a1d8-3506a8710322","Type":"ContainerStarted","Data":"af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b"}
Nov 02 22:16:01.899081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.893456    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6965-bee2c1" event={"ID":"2d530518-5a6c-4fb5-a1d8-3506a8710322","Type":"ContainerStarted","Data":"ca12800cd9899766b8d128941bc4e1ffff3bf0c2971a8514fc3d12cd7d0d68ab"}
Nov 02 22:16:01.937597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.931679    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6965-bee2c1" podStartSLOduration=1.9316598059999999 podStartE2EDuration="1.931659806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:01.924738746 +0000 UTC m=+1099.232829556" watchObservedRunningTime="2026-11-02 22:16:01.931659806 +0000 UTC m=+1099.239750612"
Nov 02 22:16:01.943489 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:01.943452    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-a8b01d\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:16:01.946737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.946704    4929 scope.go:122] "RemoveContainer" containerID="c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd"
Nov 02 22:16:01.947119 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:01.947093    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd\": container with ID starting with c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd not found: ID does not exist" containerID="c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd"
Nov 02 22:16:01.947195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.947129    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd"} err="failed to get container status \"c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd\": rpc error: code = NotFound desc = could not find container \"c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd\": container with ID starting with c345397eeecad535808cdf2ebee12a381be34ceec451180dde7d0746bd6238cd not found: ID does not exist"
Nov 02 22:16:01.985713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.985675    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:16:01.985914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:01.985733    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:16:02.695198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:02.695157    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6965-bee2c1"]
Nov 02 22:16:02.702385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:02.702340    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6965-bee2c1"]
Nov 02 22:16:02.806891 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:02.806780    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6965-a8b01d\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" pod="opendatahub/test-curl-6965-a8b01d"
Nov 02 22:16:02.814854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:02.814821    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a9a23e-22e5-4122-b5b6-e6f2dfa134b9" path="/var/lib/kubelet/pods/33a9a23e-22e5-4122-b5b6-e6f2dfa134b9/volumes"
Nov 02 22:16:04.242635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:03.959260    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6965-bee2c1" podUID="2d530518-5a6c-4fb5-a1d8-3506a8710322" containerName="test-curl-6965-bee2c1" containerID="cri-o://af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b" gracePeriod=2
Nov 02 22:16:04.242635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.195871    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-128625"
Nov 02 22:16:04.331866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.331836    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\" (UniqueName: \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\") pod \"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a\" (UID: \"a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a\") "
Nov 02 22:16:04.333802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.333767    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl" pod "a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" (UID: "a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a"). InnerVolumeSpecName "kube-api-access-xdnvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:04.434110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.434071    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdnvl\" (UniqueName: \"kubernetes.io/projected/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a-kube-api-access-xdnvl\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:04.844573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.842095    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" path="/var/lib/kubelet/pods/a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a/volumes"
Nov 02 22:16:04.968393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.968363    4929 generic.go:356] "Generic (PLEG): container finished" podID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerID="d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7" exitCode=0
Nov 02 22:16:04.968570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.968439    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerDied","Data":"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"}
Nov 02 22:16:04.969026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.969009    4929 scope.go:122] "RemoveContainer" containerID="d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"
Nov 02 22:16:04.970452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.970428    4929 generic.go:356] "Generic (PLEG): container finished" podID="84d60ac4-573a-4fb2-988b-589720c87aee" containerID="20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc" exitCode=0
Nov 02 22:16:04.970511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.970452    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerDied","Data":"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"}
Nov 02 22:16:04.970894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.970863    4929 scope.go:122] "RemoveContainer" containerID="20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"
Nov 02 22:16:04.970950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.970897    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.970995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.970963    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.971026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.971000    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.972129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972105    4929 generic.go:356] "Generic (PLEG): container finished" podID="a3ccb7e2-33bd-4c1f-bfa9-99f2d467861a" containerID="4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a" exitCode=137
Nov 02 22:16:04.972212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972158    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-128625"
Nov 02 22:16:04.972212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972193    4929 scope.go:122] "RemoveContainer" containerID="4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a"
Nov 02 22:16:04.972597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972565    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.972736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972711    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.972842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.972769    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:04.988631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.988607    4929 scope.go:122] "RemoveContainer" containerID="4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a"
Nov 02 22:16:04.988933 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:04.988915    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a\": container with ID starting with 4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a not found: ID does not exist" containerID="4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a"
Nov 02 22:16:04.989015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:04.988947    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a"} err="failed to get container status \"4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a\": rpc error: code = NotFound desc = could not find container \"4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a\": container with ID starting with 4d5c3434c50d74ae7c755fa00983b1247d3bee22af1e372577293de6d4c94a2a not found: ID does not exist"
Nov 02 22:16:06.014782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.014519    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerStarted","Data":"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"}
Nov 02 22:16:06.193522 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:06.193480    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d530518_5a6c_4fb5_a1d8_3506a8710322.slice/crio-conmon-af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 22:16:06.204623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.204578    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerStarted","Data":"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"}
Nov 02 22:16:06.284619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.280471    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:16:06.291560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.287827    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:06.291560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.287898    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:06.502605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.502575    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:06.512600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.512569    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\" (UniqueName: \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\") pod \"2d530518-5a6c-4fb5-a1d8-3506a8710322\" (UID: \"2d530518-5a6c-4fb5-a1d8-3506a8710322\") "
Nov 02 22:16:06.514653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.514630    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq" pod "2d530518-5a6c-4fb5-a1d8-3506a8710322" (UID: "2d530518-5a6c-4fb5-a1d8-3506a8710322"). InnerVolumeSpecName "kube-api-access-rrvjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:06.615368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.615331    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvjq\" (UniqueName: \"kubernetes.io/projected/2d530518-5a6c-4fb5-a1d8-3506a8710322-kube-api-access-rrvjq\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:06.705571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.702931    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:16:06.705571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.703085    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:06.705571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.703129    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:06.931256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:06.929693    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d530518-5a6c-4fb5-a1d8-3506a8710322" path="/var/lib/kubelet/pods/2d530518-5a6c-4fb5-a1d8-3506a8710322/volumes"
Nov 02 22:16:07.213992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.213905    4929 generic.go:356] "Generic (PLEG): container finished" podID="2d530518-5a6c-4fb5-a1d8-3506a8710322" containerID="af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b" exitCode=137
Nov 02 22:16:07.213992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.213965    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6965-bee2c1"
Nov 02 22:16:07.214516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.213997    4929 scope.go:122] "RemoveContainer" containerID="af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b"
Nov 02 22:16:07.224524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.224508    4929 scope.go:122] "RemoveContainer" containerID="af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b"
Nov 02 22:16:07.224780 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:07.224760    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b\": container with ID starting with af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b not found: ID does not exist" containerID="af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b"
Nov 02 22:16:07.224881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.224787    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b"} err="failed to get container status \"af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b\": rpc error: code = NotFound desc = could not find container \"af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b\": container with ID starting with af8c616a12fbc536f8b397b45127895e36562c6bbb97c6c09b392075012a746b not found: ID does not exist"
Nov 02 22:16:07.274566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.272944    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:07.274566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.273003    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:07.696264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.696222    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:07.696474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:07.696279    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:08.282559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:08.278934    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:08.282559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:08.278995    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:08.700656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:08.700617    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:08.700986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:08.700921    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:09.272559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:09.272020    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:09.272559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:09.272100    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:09.696579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:09.696521    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:09.697060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:09.696599    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:10.274134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:10.274097    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:10.274415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:10.274385    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:10.705559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:10.696896    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:10.705559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:10.696961    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:10.898317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:10.898187    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:16:11.272571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:11.269443    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:11.272571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:11.269557    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:11.697179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:11.697139    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 22:16:11.697386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:11.697197    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 22:16:12.265319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:12.265279    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 22:16:12.265785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:12.265349    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 22:16:12.703328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:12.703292    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:16:12.703999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:12.703972    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:16:12.705004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:12.704979    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:16:13.363469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:13.363429    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:16:13.368758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:13.368730    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:16:17.934420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:17.934384    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:16:17.946944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:17.946915    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:16:22.947558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:22.944625    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" podUID="0e415847-630d-47ee-8cda-523e1453744c" containerName="payload-processing" containerID="cri-o://2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac" gracePeriod=25
Nov 02 22:16:22.971576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:22.950935    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" podUID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" containerName="payload-pre-processing" containerID="cri-o://b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88" gracePeriod=25
Nov 02 22:16:23.342768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.342737    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:16:23.346371 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.346342    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:16:23.448874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.448843    4929 generic.go:356] "Generic (PLEG): container finished" podID="0e415847-630d-47ee-8cda-523e1453744c" containerID="2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac" exitCode=0
Nov 02 22:16:23.449047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.448891    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"
Nov 02 22:16:23.449047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.448926    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" event={"ID":"0e415847-630d-47ee-8cda-523e1453744c","Type":"ContainerDied","Data":"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"}
Nov 02 22:16:23.449047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.448979    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh" event={"ID":"0e415847-630d-47ee-8cda-523e1453744c","Type":"ContainerDied","Data":"6cb469212fde30008a6a7916714e98ee6890b2323874736e42b1ae6db95a6e9f"}
Nov 02 22:16:23.449047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.449004    4929 scope.go:122] "RemoveContainer" containerID="2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"
Nov 02 22:16:23.450502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.450480    4929 generic.go:356] "Generic (PLEG): container finished" podID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" containerID="b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88" exitCode=0
Nov 02 22:16:23.450638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.450557    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"
Nov 02 22:16:23.450638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.450574    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" event={"ID":"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81","Type":"ContainerDied","Data":"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"}
Nov 02 22:16:23.450638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.450636    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp" event={"ID":"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81","Type":"ContainerDied","Data":"8e16158a946a866f91756d6c79e708bfb59d07155a7e6560cdff36af49b858a2"}
Nov 02 22:16:23.460030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.460007    4929 scope.go:122] "RemoveContainer" containerID="2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"
Nov 02 22:16:23.460354 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:23.460332    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac\": container with ID starting with 2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac not found: ID does not exist" containerID="2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"
Nov 02 22:16:23.460414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.460364    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac"} err="failed to get container status \"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac\": rpc error: code = NotFound desc = could not find container \"2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac\": container with ID starting with 2a2c909968709be12c7dbd4fe3c53f8b76f325c864d18e735ecc8e7323073eac not found: ID does not exist"
Nov 02 22:16:23.460414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.460381    4929 scope.go:122] "RemoveContainer" containerID="b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"
Nov 02 22:16:23.471440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.471416    4929 scope.go:122] "RemoveContainer" containerID="b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"
Nov 02 22:16:23.471873 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:23.471853    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88\": container with ID starting with b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88 not found: ID does not exist" containerID="b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"
Nov 02 22:16:23.471946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.471882    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88"} err="failed to get container status \"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88\": rpc error: code = NotFound desc = could not find container \"b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88\": container with ID starting with b7d3cab208f8e04f7d0a7dfaec09f01bce0db9e75066fe7e1dd660c9798ccf88 not found: ID does not exist"
Nov 02 22:16:23.484924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.484891    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\" (UniqueName: \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\") pod \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") "
Nov 02 22:16:23.485102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.485083    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\") pod \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\" (UID: \"60e6dca8-b3f2-4c2d-97bf-e50c207e5e81\") "
Nov 02 22:16:23.485193 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.485181    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\") pod \"0e415847-630d-47ee-8cda-523e1453744c\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") "
Nov 02 22:16:23.485258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.485245    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\") pod \"0e415847-630d-47ee-8cda-523e1453744c\" (UID: \"0e415847-630d-47ee-8cda-523e1453744c\") "
Nov 02 22:16:23.485517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.485486    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume" pod "60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" (UID: "60e6dca8-b3f2-4c2d-97bf-e50c207e5e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:23.485689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.485545    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume" pod "0e415847-630d-47ee-8cda-523e1453744c" (UID: "0e415847-630d-47ee-8cda-523e1453744c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:23.486089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.486067    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:23.486179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.486091    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e415847-630d-47ee-8cda-523e1453744c-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:23.487256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.487236    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47" pod "60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" (UID: "60e6dca8-b3f2-4c2d-97bf-e50c207e5e81"). InnerVolumeSpecName "kube-api-access-gqr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:23.487580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.487509    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn" pod "0e415847-630d-47ee-8cda-523e1453744c" (UID: "0e415847-630d-47ee-8cda-523e1453744c"). InnerVolumeSpecName "kube-api-access-k6rqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:23.587505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.587462    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqr47\" (UniqueName: \"kubernetes.io/projected/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81-kube-api-access-gqr47\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:23.587505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.587500    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/0e415847-630d-47ee-8cda-523e1453744c-kube-api-access-k6rqn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:23.792550 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.792508    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:16:23.796750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.796722    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10852f9c-5bf9cc66bc-hv9lh"]
Nov 02 22:16:23.807072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.807032    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:16:23.812839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:23.812796    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10852f9c-69fb78bb87-ddmfp"]
Nov 02 22:16:24.807278 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:24.807200    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e415847-630d-47ee-8cda-523e1453744c" path="/var/lib/kubelet/pods/0e415847-630d-47ee-8cda-523e1453744c/volumes"
Nov 02 22:16:24.807753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:24.807738    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" path="/var/lib/kubelet/pods/60e6dca8-b3f2-4c2d-97bf-e50c207e5e81/volumes"
Nov 02 22:16:27.587483 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:27.587442    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:16:30.989222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.989187    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:16:30.990354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990327    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2d530518-5a6c-4fb5-a1d8-3506a8710322" containerName="test-curl-6965-bee2c1"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990499    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d530518-5a6c-4fb5-a1d8-3506a8710322" containerName="test-curl-6965-bee2c1"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990582    4929 cpu_manager.go:414] "containerMap: removing container" podUID="0e415847-630d-47ee-8cda-523e1453744c" containerName="payload-processing"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990592    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e415847-630d-47ee-8cda-523e1453744c" containerName="payload-processing"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990622    4929 cpu_manager.go:414] "containerMap: removing container" podUID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" containerName="payload-pre-processing"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990630    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" containerName="payload-pre-processing"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990806    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e6dca8-b3f2-4c2d-97bf-e50c207e5e81" containerName="payload-pre-processing"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990825    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d530518-5a6c-4fb5-a1d8-3506a8710322" containerName="test-curl-6965-bee2c1"
Nov 02 22:16:30.990926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.990837    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e415847-630d-47ee-8cda-523e1453744c" containerName="payload-processing"
Nov 02 22:16:30.995594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.995573    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:30.996062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.996032    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:16:30.999747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:30.999722    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-ac0910-openshift-default-dockercfg-sqz89\""
Nov 02 22:16:31.230560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230457    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.230560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230513    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.230841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230620    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcr7p\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.230841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230660    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.230841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230695    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.230841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230831    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.231049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230892    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.231049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230919    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.231049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.230948    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336473    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336570    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336598    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336628    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336821    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336849    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336940    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcr7p\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.336977    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.337012    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.337713    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.338869    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.340349    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.340878    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.384385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.341001    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.439384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.424259    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.439384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.425006    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.441133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.441096    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.456094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.455616    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcr7p\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\") pod \"e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") " pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:31.633043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:31.632961    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:32.013346 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.013264    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:16:32.065317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.065284    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:16:32.065951 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:16:32.065925    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43989402_9353_4abd_bf15_425b0381eaba.slice/crio-dd2fac835408c4233277ddf04cdee332c321eb675a77bcd61421713f5e4ffeeb WatchSource:0}: Error finding container dd2fac835408c4233277ddf04cdee332c321eb675a77bcd61421713f5e4ffeeb: Status 404 returned error can't find the container with id dd2fac835408c4233277ddf04cdee332c321eb675a77bcd61421713f5e4ffeeb
Nov 02 22:16:32.068848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.068814    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:32.068909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.068883    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:32.068948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.068912    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:32.669042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.669004    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerStarted","Data":"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"}
Nov 02 22:16:32.669243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:32.669064    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerStarted","Data":"dd2fac835408c4233277ddf04cdee332c321eb675a77bcd61421713f5e4ffeeb"}
Nov 02 22:16:33.656259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:33.648502    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:33.678882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:33.678849    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:33.722925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:33.722896    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:33.723204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:33.723191    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:16:33.774724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:33.774653    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podStartSLOduration=3.77463081 podStartE2EDuration="3.77463081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:32.718790217 +0000 UTC m=+1130.026881022" watchObservedRunningTime="2026-11-02 22:16:33.77463081 +0000 UTC m=+1131.082721613"
Nov 02 22:16:34.118594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:34.108325    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:16:37.121600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.109961    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:16:37.298854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.298753    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:16:37.299094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.298926    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.311597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.302857    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-ac0910-dockercfg-qww5d\""
Nov 02 22:16:37.311597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.305937    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-ac0910\""
Nov 02 22:16:37.407163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.405037    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.407163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.405184    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf88\" (UniqueName: \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.604560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.552846    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.608861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.571976    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.704611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.680448    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf88\" (UniqueName: \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.747916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.736268    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:16:37.763577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.752619    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf88\" (UniqueName: \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\") pod \"payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:37.803570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.802990    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:37.831175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.827895    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:16:37.869854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:37.857393    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:38.022064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.022031    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxjv\" (UniqueName: \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.022508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.022471    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.159558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.158982    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxjv\" (UniqueName: \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.159558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.159181    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.161054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.160329    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.194979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.190442    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxjv\" (UniqueName: \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\") pod \"payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.256603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.249865    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:38.833887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.833863    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:16:38.842047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:38.842023    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:16:39.367064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.367014    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" event={"ID":"f20930ed-6381-4551-8423-e9a0abfab58d","Type":"ContainerStarted","Data":"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"}
Nov 02 22:16:39.367522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.367135    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:39.367522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.367159    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" event={"ID":"f20930ed-6381-4551-8423-e9a0abfab58d","Type":"ContainerStarted","Data":"e3e43f6c3cf1af2420ffccbda061ad0c85ef4bc656f6863ca7abb5bc52fdd8c4"}
Nov 02 22:16:39.369388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.369343    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" event={"ID":"21e473fb-93fc-4e27-9d9e-241ea3ea9b60","Type":"ContainerStarted","Data":"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"}
Nov 02 22:16:39.369508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.369401    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" event={"ID":"21e473fb-93fc-4e27-9d9e-241ea3ea9b60","Type":"ContainerStarted","Data":"7f557f8d4b806c5f361fa73ba879d21bcfa8f70408cc3d34f9c6fa6896c7b1eb"}
Nov 02 22:16:39.369508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.369436    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:39.388040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.387960    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" podStartSLOduration=3.387946963 podStartE2EDuration="3.387946963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:39.382791139 +0000 UTC m=+1136.690881944" watchObservedRunningTime="2026-11-02 22:16:39.387946963 +0000 UTC m=+1136.696037768"
Nov 02 22:16:39.402659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:39.402607    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" podStartSLOduration=3.402592887 podStartE2EDuration="3.402592887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:39.398164569 +0000 UTC m=+1136.706255371" watchObservedRunningTime="2026-11-02 22:16:39.402592887 +0000 UTC m=+1136.710683692"
Nov 02 22:16:40.838975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.838927    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:16:40.845435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.845407    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.860062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.860026    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:16:40.870170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.870104    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2fc3ce-openshift-default-dockercfg-8vm4z\""
Nov 02 22:16:40.982134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982087    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982138    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9s2\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982278    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982341    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982408    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982452    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982641    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982716    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:40.982770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:40.982741    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093362    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093422    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093518    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093566    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9s2\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093681    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093825    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093861    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.093963    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.094213    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.094785    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.095057    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.095287    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.096698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.095513    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.097664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.097347    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.124558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.118335    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.139894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.129110    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.139894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.129854    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9s2\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.139894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.130077    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.181160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.180868    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:41.900482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.900439    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:16:41.912323 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.912266    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:41.912471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.912425    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:41.912588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:41.912516    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:16:42.428952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:42.428910    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerStarted","Data":"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"}
Nov 02 22:16:42.429138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:42.428971    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerStarted","Data":"d0d987f0964ef35a861a9a7024cc5a347028997e3ef740236a087bff721d0acf"}
Nov 02 22:16:42.452875 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:42.452817    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podStartSLOduration=2.452799886 podStartE2EDuration="2.452799886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:42.447519823 +0000 UTC m=+1139.755610665" watchObservedRunningTime="2026-11-02 22:16:42.452799886 +0000 UTC m=+1139.760890690"
Nov 02 22:16:43.194558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:43.189000    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:43.222659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:43.218118    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:43.438497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:43.438456    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:43.439454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:43.439433    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:16:46.031163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.031117    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:16:46.038341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.038307    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.039136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.039104    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:16:46.049125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.049025    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2fc3ce\""
Nov 02 22:16:46.050779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.050751    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2fc3ce-dockercfg-6spdv\""
Nov 02 22:16:46.165428 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.165377    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxht\" (UniqueName: \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.165775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.165735    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.303318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.303212    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxht\" (UniqueName: \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.304556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.304510    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.305747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.305636    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.339561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.330851    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxht\" (UniqueName: \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\") pod \"payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.374685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.366936    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:46.880146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:46.879906    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:16:46.918030 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:16:46.917990    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856dae52_ce56_4b29_a210_e391c6d35ad5.slice/crio-d6bf48a7b70ea227f7fbd9443f81c0f307e07dc3f42135df24134d36803a559c WatchSource:0}: Error finding container d6bf48a7b70ea227f7fbd9443f81c0f307e07dc3f42135df24134d36803a559c: Status 404 returned error can't find the container with id d6bf48a7b70ea227f7fbd9443f81c0f307e07dc3f42135df24134d36803a559c
Nov 02 22:16:47.566621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:47.566582    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" event={"ID":"856dae52-ce56-4b29-a210-e391c6d35ad5","Type":"ContainerStarted","Data":"222dfd75fbb0251fad35c453f30893ca5cb4bcced39b9b0284d19dff3a915a52"}
Nov 02 22:16:47.567129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:47.566640    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" event={"ID":"856dae52-ce56-4b29-a210-e391c6d35ad5","Type":"ContainerStarted","Data":"d6bf48a7b70ea227f7fbd9443f81c0f307e07dc3f42135df24134d36803a559c"}
Nov 02 22:16:47.567702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:47.567678    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:16:47.601422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:47.588270    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" podStartSLOduration=2.588252926 podStartE2EDuration="2.588252926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:47.582687989 +0000 UTC m=+1144.890778797" watchObservedRunningTime="2026-11-02 22:16:47.588252926 +0000 UTC m=+1144.896343731"
Nov 02 22:16:50.380486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:50.380455    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:16:50.380917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:50.380522    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:16:57.593199 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:16:57.590823    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:16:59.617056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:16:59.617021    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:17:04.581980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.581938    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6971-fee234"]
Nov 02 22:17:04.587112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.587080    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:04.591761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.591730    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-fee234"]
Nov 02 22:17:04.592002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.591981    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8p5jl\""
Nov 02 22:17:04.592090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.592016    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:17:04.592090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.592016    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:17:04.747871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.747512    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgktd\" (UniqueName: \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\") pod \"test-curl-6971-fee234\" (UID: \"236e41ee-e7f7-4db2-b790-22dd9d4e38ea\") " pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:04.850656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.849501    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgktd\" (UniqueName: \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\") pod \"test-curl-6971-fee234\" (UID: \"236e41ee-e7f7-4db2-b790-22dd9d4e38ea\") " pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:04.858422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.858390    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgktd\" (UniqueName: \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\") pod \"test-curl-6971-fee234\" (UID: \"236e41ee-e7f7-4db2-b790-22dd9d4e38ea\") " pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:04.916237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:04.915856    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:05.298230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:05.298190    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-fee234"]
Nov 02 22:17:05.298828 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:17:05.298799    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod236e41ee_e7f7_4db2_b790_22dd9d4e38ea.slice/crio-5be615966f939a459baab991c29a22a6e39a4afcd5025f36102181af002bf3bd WatchSource:0}: Error finding container 5be615966f939a459baab991c29a22a6e39a4afcd5025f36102181af002bf3bd: Status 404 returned error can't find the container with id 5be615966f939a459baab991c29a22a6e39a4afcd5025f36102181af002bf3bd
Nov 02 22:17:06.353254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:06.352414    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-fee234" event={"ID":"236e41ee-e7f7-4db2-b790-22dd9d4e38ea","Type":"ContainerStarted","Data":"10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7"}
Nov 02 22:17:06.353254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:06.352468    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-fee234" event={"ID":"236e41ee-e7f7-4db2-b790-22dd9d4e38ea","Type":"ContainerStarted","Data":"5be615966f939a459baab991c29a22a6e39a4afcd5025f36102181af002bf3bd"}
Nov 02 22:17:06.367276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:06.367217    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6971-fee234" podStartSLOduration=2.367202 podStartE2EDuration="2.367202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:06.366290585 +0000 UTC m=+1163.674381389" watchObservedRunningTime="2026-11-02 22:17:06.367202 +0000 UTC m=+1163.675292804"
Nov 02 22:17:07.061693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:07.061636    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6971-fee234"]
Nov 02 22:17:07.066394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:07.066361    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6971-fee234"]
Nov 02 22:17:08.402187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:08.402144    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6971-fee234" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" containerName="test-curl-6971-fee234" containerID="cri-o://10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7" gracePeriod=2
Nov 02 22:17:08.404612 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:08.404564    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-fee234\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:10.923681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:10.923647    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:11.035528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.035491    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\" (UniqueName: \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\") pod \"236e41ee-e7f7-4db2-b790-22dd9d4e38ea\" (UID: \"236e41ee-e7f7-4db2-b790-22dd9d4e38ea\") "
Nov 02 22:17:11.058905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.055312    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd" pod "236e41ee-e7f7-4db2-b790-22dd9d4e38ea" (UID: "236e41ee-e7f7-4db2-b790-22dd9d4e38ea"). InnerVolumeSpecName "kube-api-access-jgktd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:11.138375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.138342    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgktd\" (UniqueName: \"kubernetes.io/projected/236e41ee-e7f7-4db2-b790-22dd9d4e38ea-kube-api-access-jgktd\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:11.234900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.234853    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:17:11.235822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.235801    4929 cpu_manager.go:414] "containerMap: removing container" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" containerName="test-curl-6971-fee234"
Nov 02 22:17:11.235916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.235823    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" containerName="test-curl-6971-fee234"
Nov 02 22:17:11.235964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.235954    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" containerName="test-curl-6971-fee234"
Nov 02 22:17:11.239818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.239793    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.254410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.254368    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0dd701\""
Nov 02 22:17:11.255829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.254942    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0dd701-dockercfg-4hmqr\""
Nov 02 22:17:11.259318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.259289    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:17:11.362561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.358508    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ls5\" (UniqueName: \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.362561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.358634    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.477582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.465839    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ls5\" (UniqueName: \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.477582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.465971    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.477582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.467398    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.551898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.551704    4929 generic.go:356] "Generic (PLEG): container finished" podID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" containerID="10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7" exitCode=137
Nov 02 22:17:11.551898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.551844    4929 scope.go:122] "RemoveContainer" containerID="10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7"
Nov 02 22:17:11.566578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.552296    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:11.566578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.555204    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ls5\" (UniqueName: \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\") pod \"payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") " pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.568308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.568277    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:11.756147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.756109    4929 scope.go:122] "RemoveContainer" containerID="10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7"
Nov 02 22:17:11.782643 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:11.756570    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7\": container with ID starting with 10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7 not found: ID does not exist" containerID="10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7"
Nov 02 22:17:11.782643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:11.756613    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7"} err="failed to get container status \"10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7\": rpc error: code = NotFound desc = could not find container \"10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7\": container with ID starting with 10a06a9b626c507c5119d8bb6d9809b406ca8144e0f9fa65488dce0e007c6ac7 not found: ID does not exist"
Nov 02 22:17:11.782643 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:11.761998    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-fee234\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:12.120631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.115107    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:17:12.562239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.562141    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" event={"ID":"a0148602-65e7-426a-8344-3b9d162bd626","Type":"ContainerStarted","Data":"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"}
Nov 02 22:17:12.562239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.562202    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" event={"ID":"a0148602-65e7-426a-8344-3b9d162bd626","Type":"ContainerStarted","Data":"396f56a70336afbbfd5514d8e090711286f20cd524919e9d1a034d1ca2d46749"}
Nov 02 22:17:12.562239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.562223    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:12.564721 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:12.564695    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-fee234\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:12.584161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.584101    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" podStartSLOduration=1.584086819 podStartE2EDuration="1.584086819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:12.579837988 +0000 UTC m=+1169.887928795" watchObservedRunningTime="2026-11-02 22:17:12.584086819 +0000 UTC m=+1169.892177623"
Nov 02 22:17:12.817764 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:12.815764    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-fee234\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" pod="opendatahub/test-curl-6971-fee234"
Nov 02 22:17:12.876946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:12.876119    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236e41ee-e7f7-4db2-b790-22dd9d4e38ea" path="/var/lib/kubelet/pods/236e41ee-e7f7-4db2-b790-22dd9d4e38ea/volumes"
Nov 02 22:17:14.167601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:14.167567    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:17:18.346014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.345974    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:17:18.428792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.428736    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:17:18.499720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.499669    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:18.528705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.507626    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:18.815564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.809576    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 22:17:18.829243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.829194    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:18.855608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.844464    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:18.856238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.844315    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:18.856577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:18.855877    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:19.062558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.057045    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:17:19.095663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.087705    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:17:19.111561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.095882    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:19.137288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.113082    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:19.145174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.141339    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:17:19.509695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.509643    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:19.510875 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.510837    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:17:19.555290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.555224    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:17:19.622559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.584093    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:17:19.829460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.828719    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:17:19.861837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:19.861636    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.011003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.010948    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:17:20.011695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.011662    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:17:20.033003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.032969    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:17:20.033367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.033330    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:17:20.033720 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.033701    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:17:20.059759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.058553    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.062262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.062229    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:17:20.068159 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.067217    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.071614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.069422    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:17:20.175003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.170046    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.177837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.177778    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:17:20.180266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.180135    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-j6cnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:17:20.180266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.180211    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-j6cnb" podUID="047c4566-edc7-453d-bd8c-d14867259b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.209586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.209125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:17:20.209586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.209446    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:17:20.373043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.371485    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.706557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.636181    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.755567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.746426    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:17:20.786394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.786336    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 22:17:20.922825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.922775    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:17:20.985001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.934816    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:20.985001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:20.944454    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.051094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.006775    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.098255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.071832    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" podUID="7a4a8e4d-2131-4854-9d0b-5010dae227bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.098255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.082489    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.098255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.083150    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.098727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.098678    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.185057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.176696    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.213716 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.213640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:21.484561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.475108    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:21.691561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.688130    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:17:21.780622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.759981    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:17:21.866390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.866348    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:17:21.958958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:21.893076    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:17:22.284785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.020867    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.285313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.022130    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:17:22.311119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.025665    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:17:22.315121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.025710    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 02 22:17:22.336833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.025735    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:17:22.349601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.171314    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.349849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.190472    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.350903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.193828    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.355607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.193860    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:17:22.356915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.264761    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.490746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.490700    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 02 22:17:22.565307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.501705    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:22.672561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.662267    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:17:22.672561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.664316    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:17:22.719909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.719863    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:17:22.722122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.722084    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:17:22.777787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.777755    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:17:22.780121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:22.780094    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:17:23.194714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.192162    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Nov 02 22:17:23.317558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.312020    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:23.317558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.312198    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:17:23.399603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.349680    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing" containerID="cri-o://af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d" gracePeriod=25
Nov 02 22:17:23.527758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.355650    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:17:23.537816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.359648    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:17:23.548563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.429682    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing" containerID="cri-o://091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c" gracePeriod=25
Nov 02 22:17:23.570563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.569621    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded"
Nov 02 22:17:23.589562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.579937    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:17:23.589562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.580004    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:23.693561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.685098    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:17:23.694081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.694050    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:17:23.718559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.704172    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:17:23.829165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.827850    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:17:23.933837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.930582    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:17:23.967799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:23.967765    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:17:25.947700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:25.947403    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:17:25.953944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:25.952991    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:17:26.760900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.760859    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:17:26.778812 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.778776    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.782948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.782905    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d9416f-openshift-default-dockercfg-46hpf\""
Nov 02 22:17:26.785333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.785219    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:17:26.830678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830633    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.830921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830687    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.830921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830719    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.830921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830834    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.830921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830865    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl64m\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.830921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.830910    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.831181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.831066    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.831181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.831164    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.831284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.831274    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.957641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.944053    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.964372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.958267    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl64m\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.964843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.964803    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.973559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.965192    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.974049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.973946    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.974525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.974499    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:26.988560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:26.974802    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.043457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.043351    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.043876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.043844    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.044949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.044908    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.056600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.048922    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.127569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.124404    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.127569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.125049    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.149562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.138346    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.197451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.177799    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl64m\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.234133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.214954    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.234133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.217849    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.293765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.278787    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\") pod \"e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") " pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.426970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.426936    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:27.628914 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:27.628878    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:17:27.880190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.880159    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:17:27.881518 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:17:27.881489    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fa532d_5c35_49ee_97b1_7b21d30956b3.slice/crio-44c95a3b1e6110508cc382d73ca9b402e234f1c102ba7224f1805da40b5d8957 WatchSource:0}: Error finding container 44c95a3b1e6110508cc382d73ca9b402e234f1c102ba7224f1805da40b5d8957: Status 404 returned error can't find the container with id 44c95a3b1e6110508cc382d73ca9b402e234f1c102ba7224f1805da40b5d8957
Nov 02 22:17:27.884771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.884737    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:27.884838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.884811    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:27.884884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:27.884841    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:28.909941 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:28.909207    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podStartSLOduration=2.909180631 podStartE2EDuration="2.909180631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:28.900015845 +0000 UTC m=+1186.208106671" watchObservedRunningTime="2026-11-02 22:17:28.909180631 +0000 UTC m=+1186.217271469"
Nov 02 22:17:28.925790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:28.925743    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerStarted","Data":"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"}
Nov 02 22:17:28.925986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:28.925840    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerStarted","Data":"44c95a3b1e6110508cc382d73ca9b402e234f1c102ba7224f1805da40b5d8957"}
Nov 02 22:17:29.428273 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.428230    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:29.434603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.434566    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:29.480754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.480706    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:17:29.480937 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.480778    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:17:29.892525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.889389    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:29.892525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:29.892022    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:17:32.585817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.585735    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:17:32.590702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.590675    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.595241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.595184    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d9416f\""
Nov 02 22:17:32.599419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.599394    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:17:32.613789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.613740    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:17:32.619591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.619564    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.637455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.637032    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d9416f-dockercfg-nvhg8\""
Nov 02 22:17:32.637714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.637682    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:17:32.721423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.720772    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6khc\" (UniqueName: \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.721423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.720843    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.721423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.720963    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.721423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.721046    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjm4m\" (UniqueName: \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.848423    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.848559    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm4m\" (UniqueName: \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.848914    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6khc\" (UniqueName: \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.848967    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.850094    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.855611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.850893    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.882558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.875327    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm4m\" (UniqueName: \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\") pod \"payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") " pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.898559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.889783    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6khc\" (UniqueName: \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\") pod \"payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:32.908579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.907855    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:32.971165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:32.971115    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:33.135156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.134956    4929 generic.go:356] "Generic (PLEG): container finished" podID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerID="73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792" exitCode=0
Nov 02 22:17:33.135802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.135678    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerDied","Data":"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"}
Nov 02 22:17:33.164557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.151099    4929 scope.go:122] "RemoveContainer" containerID="73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"
Nov 02 22:17:33.194077 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.194002    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.196892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.196852    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.197140 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.197116    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.204392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.204325    4929 generic.go:356] "Generic (PLEG): container finished" podID="43989402-9353-4abd-bf15-425b0381eaba" containerID="8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d" exitCode=0
Nov 02 22:17:33.214196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.211057    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerDied","Data":"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"}
Nov 02 22:17:33.214196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.211710    4929 scope.go:122] "RemoveContainer" containerID="8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"
Nov 02 22:17:33.234408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.234366    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.234611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.234458    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.234611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.234504    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:33.683274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.683230    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:17:33.735661 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.728806    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:17:33.810699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:33.810052    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:17:35.031072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.026677    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerStarted","Data":"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"}
Nov 02 22:17:35.031586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.031422    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerStarted","Data":"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"}
Nov 02 22:17:35.035555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.035141    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" event={"ID":"bb1fa85d-e0bb-4609-b720-4ca63d8e5460","Type":"ContainerStarted","Data":"073aa473221ef348f094f7aaff740585ff315b09773bc06eea8d1b8398f6beae"}
Nov 02 22:17:35.035555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.035191    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" event={"ID":"bb1fa85d-e0bb-4609-b720-4ca63d8e5460","Type":"ContainerStarted","Data":"f2886a26327df592ed2739b2760d7f1c129685f44588296bd59a62d2f70df40d"}
Nov 02 22:17:35.037185 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.037128    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:35.040376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.039420    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" event={"ID":"3fcc68a1-888c-4dc2-887f-acbbb52cb850","Type":"ContainerStarted","Data":"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"}
Nov 02 22:17:35.040376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.039468    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" event={"ID":"3fcc68a1-888c-4dc2-887f-acbbb52cb850","Type":"ContainerStarted","Data":"40d72038ba0de9c13b83a268cc424e8e47d82aaf475cf8b033974563ecfc0c52"}
Nov 02 22:17:35.040376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.040318    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:35.125108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.125051    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" podStartSLOduration=3.12503521 podStartE2EDuration="3.12503521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:35.118135518 +0000 UTC m=+1192.426226323" watchObservedRunningTime="2026-11-02 22:17:35.12503521 +0000 UTC m=+1192.433126016"
Nov 02 22:17:35.196252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.194843    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:35.196252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.194913    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:35.196252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.194946    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:17:35.634204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.634171    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:17:35.665562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.664683    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:35.665562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.664753    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:35.993872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:35.993770    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:17:36.004099 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:36.004057    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:17:36.228610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:36.228402    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:36.228610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:36.228492    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:36.634345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:36.634311    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:36.634605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:36.634386    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:37.182467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:37.182419    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:37.182771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:37.182505    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:37.634265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:37.634232    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:37.634746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:37.634300    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:38.189557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:38.181759    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:38.189557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:38.181843    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:38.661558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:38.661053    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:38.661558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:38.661118    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:39.188573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:39.186423    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:39.188573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:39.186485    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:39.667559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:39.655015    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:39.667559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:39.655074    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:40.205584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:40.205526    4929 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 02 22:17:40.205877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:40.205847    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 02 22:17:40.649570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:40.634582    4929 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 22:17:40.649570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:40.634649    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 22:17:41.188501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.188470    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:17:41.188974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.188856    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:17:41.189705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.189680    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:17:41.215047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.214995    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" podStartSLOduration=9.214981116 podStartE2EDuration="9.214981116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:35.195459472 +0000 UTC m=+1192.503550265" watchObservedRunningTime="2026-11-02 22:17:41.214981116 +0000 UTC m=+1198.523071914"
Nov 02 22:17:41.650905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.650865    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:17:41.651653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.651629    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:17:41.653690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:41.653654    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:17:43.967197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:43.967165    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:17:44.052142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.052112    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:17:44.053176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.053142    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\" (UniqueName: \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\") pod \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") "
Nov 02 22:17:44.053426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.053403    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\") pod \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\" (UID: \"3d6efa77-19ef-46cb-b642-0f67b0e294dd\") "
Nov 02 22:17:44.053825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.053794    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume" pod "3d6efa77-19ef-46cb-b642-0f67b0e294dd" (UID: "3d6efa77-19ef-46cb-b642-0f67b0e294dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:44.054470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.054434    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6efa77-19ef-46cb-b642-0f67b0e294dd-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:44.055950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.055922    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz" pod "3d6efa77-19ef-46cb-b642-0f67b0e294dd" (UID: "3d6efa77-19ef-46cb-b642-0f67b0e294dd"). InnerVolumeSpecName "kube-api-access-dz4dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:44.155992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.155941    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\") pod \"74fc6993-bf61-499d-bf3e-4654cdda1cee\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") "
Nov 02 22:17:44.156169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.156071    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\" (UniqueName: \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\") pod \"74fc6993-bf61-499d-bf3e-4654cdda1cee\" (UID: \"74fc6993-bf61-499d-bf3e-4654cdda1cee\") "
Nov 02 22:17:44.156415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.156387    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume" pod "74fc6993-bf61-499d-bf3e-4654cdda1cee" (UID: "74fc6993-bf61-499d-bf3e-4654cdda1cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:44.157216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.157191    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4dz\" (UniqueName: \"kubernetes.io/projected/3d6efa77-19ef-46cb-b642-0f67b0e294dd-kube-api-access-dz4dz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:44.157271 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.157221    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fc6993-bf61-499d-bf3e-4654cdda1cee-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:44.158289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.158264    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w" pod "74fc6993-bf61-499d-bf3e-4654cdda1cee" (UID: "74fc6993-bf61-499d-bf3e-4654cdda1cee"). InnerVolumeSpecName "kube-api-access-ws98w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:44.259499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.259411    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws98w\" (UniqueName: \"kubernetes.io/projected/74fc6993-bf61-499d-bf3e-4654cdda1cee-kube-api-access-ws98w\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:44.613281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.613245    4929 generic.go:356] "Generic (PLEG): container finished" podID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerID="091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c" exitCode=0
Nov 02 22:17:44.613506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.613323    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"
Nov 02 22:17:44.613506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.613336    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" event={"ID":"74fc6993-bf61-499d-bf3e-4654cdda1cee","Type":"ContainerDied","Data":"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"}
Nov 02 22:17:44.613506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.613404    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx" event={"ID":"74fc6993-bf61-499d-bf3e-4654cdda1cee","Type":"ContainerDied","Data":"5aa4706894ac067bc4adc93304dfcc4cb6fb82ea1ce42d1119636da48fa49a94"}
Nov 02 22:17:44.613506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.613436    4929 scope.go:122] "RemoveContainer" containerID="091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"
Nov 02 22:17:44.615789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.615761    4929 generic.go:356] "Generic (PLEG): container finished" podID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerID="af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d" exitCode=0
Nov 02 22:17:44.615998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.615959    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" event={"ID":"3d6efa77-19ef-46cb-b642-0f67b0e294dd","Type":"ContainerDied","Data":"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"}
Nov 02 22:17:44.616047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.616011    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff" event={"ID":"3d6efa77-19ef-46cb-b642-0f67b0e294dd","Type":"ContainerDied","Data":"4722eba8ad1a07463fce66f899bc0c546b1aa9b3438583deff425a0d60e0b57a"}
Nov 02 22:17:44.616098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.616089    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"
Nov 02 22:17:44.626605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.626583    4929 scope.go:122] "RemoveContainer" containerID="091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"
Nov 02 22:17:44.626925 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:44.626906    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c\": container with ID starting with 091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c not found: ID does not exist" containerID="091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"
Nov 02 22:17:44.626979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.626935    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c"} err="failed to get container status \"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c\": rpc error: code = NotFound desc = could not find container \"091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c\": container with ID starting with 091f7b71263b8d54f643bfcc474318c80b6d035779409b744dda85818a02823c not found: ID does not exist"
Nov 02 22:17:44.626979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.626951    4929 scope.go:122] "RemoveContainer" containerID="af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"
Nov 02 22:17:44.638872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.638847    4929 scope.go:122] "RemoveContainer" containerID="af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"
Nov 02 22:17:44.639214 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:44.639193    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d\": container with ID starting with af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d not found: ID does not exist" containerID="af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"
Nov 02 22:17:44.639290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.639223    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d"} err="failed to get container status \"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d\": rpc error: code = NotFound desc = could not find container \"af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d\": container with ID starting with af171e2099018311035c07030b9767a9ffdb8c1e7938c09a3dfb24497d381f4d not found: ID does not exist"
Nov 02 22:17:44.644052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.644027    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:17:44.648602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.648575    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-4nfkx"]
Nov 02 22:17:44.655203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.655169    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:17:44.659623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.659594    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-czmff"]
Nov 02 22:17:44.812486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.812450    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" path="/var/lib/kubelet/pods/3d6efa77-19ef-46cb-b642-0f67b0e294dd/volumes"
Nov 02 22:17:44.813222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:44.813190    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" path="/var/lib/kubelet/pods/74fc6993-bf61-499d-bf3e-4654cdda1cee/volumes"
Nov 02 22:17:45.466937 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:45.466905    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:17:45.470181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:45.470157    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:17:47.454568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:47.454298    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:17:47.456309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:47.456278    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:17:47.581697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:47.579266    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:17:50.074305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.074256    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:17:50.075155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075138    4929 cpu_manager.go:414] "containerMap: removing container" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing"
Nov 02 22:17:50.075200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075156    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing"
Nov 02 22:17:50.075200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075186    4929 cpu_manager.go:414] "containerMap: removing container" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing"
Nov 02 22:17:50.075200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075191    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing"
Nov 02 22:17:50.075766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075310    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fc6993-bf61-499d-bf3e-4654cdda1cee" containerName="payload-pre-processing"
Nov 02 22:17:50.075766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.075325    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6efa77-19ef-46cb-b642-0f67b0e294dd" containerName="payload-processing"
Nov 02 22:17:50.081145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.081115    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.087287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.087002    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-84a94060-dockercfg-cbs65\""
Nov 02 22:17:50.087287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.087090    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-84a94060\""
Nov 02 22:17:50.087287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.087247    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:17:50.112823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.112785    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:17:50.119473 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.119439    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.121762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.121730    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:17:50.219557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.218983    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9mf\" (UniqueName: \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.219557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.219227    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.219557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.219350    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.219557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.219386    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb2h\" (UniqueName: \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.337634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.337006    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.337634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.337123    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.337634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.337155    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb2h\" (UniqueName: \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.337634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.337485    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9mf\" (UniqueName: \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.421561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.420624    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.471564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.465479    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.642761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.637289    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9mf\" (UniqueName: \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\") pod \"payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:50.670422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.670230    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb2h\" (UniqueName: \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\") pod \"payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") " pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.708225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.708182    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:50.740601 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:50.740502    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:51.125494 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:17:51.125462    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2997319_b8a0_464e_b791_59b4bb8537c1.slice/crio-9347c2e412b9cd5cae750285a5f8121d46f9a102f7288b0291e63a68d7ede366 WatchSource:0}: Error finding container 9347c2e412b9cd5cae750285a5f8121d46f9a102f7288b0291e63a68d7ede366: Status 404 returned error can't find the container with id 9347c2e412b9cd5cae750285a5f8121d46f9a102f7288b0291e63a68d7ede366
Nov 02 22:17:51.125862 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.125738    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:17:51.151283 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:17:51.151244    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice/crio-e3dedbe81fe8c074d5072b1cf03b0f6653d60653e723789111fd3e9aff337582 WatchSource:0}: Error finding container e3dedbe81fe8c074d5072b1cf03b0f6653d60653e723789111fd3e9aff337582: Status 404 returned error can't find the container with id e3dedbe81fe8c074d5072b1cf03b0f6653d60653e723789111fd3e9aff337582
Nov 02 22:17:51.152453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.152430    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:17:51.743218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.743177    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" event={"ID":"f04565c1-4caa-4c23-b750-4cf05ffdcf90","Type":"ContainerStarted","Data":"51b2e42734f777eeebf6aaa7bc0056dac74a8d3723a2d995b6e20349876ed20e"}
Nov 02 22:17:51.743424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.743248    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:17:51.743424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.743273    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" event={"ID":"f04565c1-4caa-4c23-b750-4cf05ffdcf90","Type":"ContainerStarted","Data":"e3dedbe81fe8c074d5072b1cf03b0f6653d60653e723789111fd3e9aff337582"}
Nov 02 22:17:51.745415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.745387    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" event={"ID":"f2997319-b8a0-464e-b791-59b4bb8537c1","Type":"ContainerStarted","Data":"696d4acf16f6a563cdb267024e3b90d4b02f58731da227004aeb8ac3a5675aac"}
Nov 02 22:17:51.745555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.745436    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" event={"ID":"f2997319-b8a0-464e-b791-59b4bb8537c1","Type":"ContainerStarted","Data":"9347c2e412b9cd5cae750285a5f8121d46f9a102f7288b0291e63a68d7ede366"}
Nov 02 22:17:51.745555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.745461    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:17:51.763418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.763364    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" podStartSLOduration=1.763349295 podStartE2EDuration="1.763349295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:51.759267728 +0000 UTC m=+1209.067358532" watchObservedRunningTime="2026-11-02 22:17:51.763349295 +0000 UTC m=+1209.071440100"
Nov 02 22:17:51.777865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:51.777810    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" podStartSLOduration=1.777795411 podStartE2EDuration="1.777795411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:51.774010177 +0000 UTC m=+1209.082100983" watchObservedRunningTime="2026-11-02 22:17:51.777795411 +0000 UTC m=+1209.085886215"
Nov 02 22:17:52.580103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:52.580057    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" podUID="856dae52-ce56-4b29-a210-e391c6d35ad5" containerName="payload-pre-processing" containerID="cri-o://222dfd75fbb0251fad35c453f30893ca5cb4bcced39b9b0284d19dff3a915a52" gracePeriod=25
Nov 02 22:17:52.757553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:52.757508    4929 generic.go:356] "Generic (PLEG): container finished" podID="856dae52-ce56-4b29-a210-e391c6d35ad5" containerID="222dfd75fbb0251fad35c453f30893ca5cb4bcced39b9b0284d19dff3a915a52" exitCode=0
Nov 02 22:17:52.757745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:52.757602    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" event={"ID":"856dae52-ce56-4b29-a210-e391c6d35ad5","Type":"ContainerDied","Data":"222dfd75fbb0251fad35c453f30893ca5cb4bcced39b9b0284d19dff3a915a52"}
Nov 02 22:17:52.940350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:52.940319    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:17:53.075558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.069500    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\") pod \"856dae52-ce56-4b29-a210-e391c6d35ad5\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") "
Nov 02 22:17:53.075558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.069862    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\" (UniqueName: \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\") pod \"856dae52-ce56-4b29-a210-e391c6d35ad5\" (UID: \"856dae52-ce56-4b29-a210-e391c6d35ad5\") "
Nov 02 22:17:53.089584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.076635    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume" pod "856dae52-ce56-4b29-a210-e391c6d35ad5" (UID: "856dae52-ce56-4b29-a210-e391c6d35ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:53.089584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.077246    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856dae52-ce56-4b29-a210-e391c6d35ad5-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:53.089584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.081628    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht" pod "856dae52-ce56-4b29-a210-e391c6d35ad5" (UID: "856dae52-ce56-4b29-a210-e391c6d35ad5"). InnerVolumeSpecName "kube-api-access-zvxht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:53.196691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.183925    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxht\" (UniqueName: \"kubernetes.io/projected/856dae52-ce56-4b29-a210-e391c6d35ad5-kube-api-access-zvxht\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:53.767591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.767559    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"
Nov 02 22:17:53.768146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.767600    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq" event={"ID":"856dae52-ce56-4b29-a210-e391c6d35ad5","Type":"ContainerDied","Data":"d6bf48a7b70ea227f7fbd9443f81c0f307e07dc3f42135df24134d36803a559c"}
Nov 02 22:17:53.768146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.767681    4929 scope.go:122] "RemoveContainer" containerID="222dfd75fbb0251fad35c453f30893ca5cb4bcced39b9b0284d19dff3a915a52"
Nov 02 22:17:53.796150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.796112    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:17:53.798838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:53.798811    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2fc3ce-5df5479b6c-jpqwq"]
Nov 02 22:17:54.942979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:54.942143    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856dae52-ce56-4b29-a210-e391c6d35ad5" path="/var/lib/kubelet/pods/856dae52-ce56-4b29-a210-e391c6d35ad5/volumes"
Nov 02 22:17:57.530556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:57.529769    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:17:57.540723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:57.531439    4929 cpu_manager.go:414] "containerMap: removing container" podUID="856dae52-ce56-4b29-a210-e391c6d35ad5" containerName="payload-pre-processing"
Nov 02 22:17:57.540723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:57.531471    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856dae52-ce56-4b29-a210-e391c6d35ad5" containerName="payload-pre-processing"
Nov 02 22:17:57.540723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:57.531789    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="856dae52-ce56-4b29-a210-e391c6d35ad5" containerName="payload-pre-processing"
Nov 02 22:17:57.708563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:57.708351    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.170606 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:17:58.170528    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:17:58.318811 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.202162    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14dae2-openshift-default-dockercfg-pnwkg\""
Nov 02 22:17:58.387653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.361734    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:17:58.678563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.668830    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709046    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709161    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709256    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709303    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709437    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709634    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709695    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxhr\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.715972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.709767    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.821872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.821798    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.823999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.823959    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.879422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.879389    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.879824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.879796    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.880145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.880123    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.880297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.880282    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxhr\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.880442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.880429    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.881085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.881054    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.881469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.881446    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.881904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.839141    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.883307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.883281    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.892567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.887691    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.894210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.894176    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.903124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.903094    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.969678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.912053    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.969678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.912199    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxhr\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.975921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.970639    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:58.979253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:58.979221    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\") pod \"e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") " pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:59.141560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.141488    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:17:59.500747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.500713    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:17:59.503267 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:17:59.503235    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f87b95_4f7f_4c53_863a_6a73687c12eb.slice/crio-a49d483bc44e95f6b02c3e7e214a75f8322d36403c292887619ed9ae6de08e40 WatchSource:0}: Error finding container a49d483bc44e95f6b02c3e7e214a75f8322d36403c292887619ed9ae6de08e40: Status 404 returned error can't find the container with id a49d483bc44e95f6b02c3e7e214a75f8322d36403c292887619ed9ae6de08e40
Nov 02 22:17:59.506710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.506671    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:59.506851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.506757    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:59.506851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.506791    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:17:59.802116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.802009    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerStarted","Data":"d42dac41d7c4a660ebc0db959be99516219fc984ea974b74f4b96cd006b89221"}
Nov 02 22:17:59.802116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.802066    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerStarted","Data":"a49d483bc44e95f6b02c3e7e214a75f8322d36403c292887619ed9ae6de08e40"}
Nov 02 22:17:59.836163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:17:59.836086    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podStartSLOduration=2.836064693 podStartE2EDuration="2.836064693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:59.833906169 +0000 UTC m=+1217.141996975" watchObservedRunningTime="2026-11-02 22:17:59.836064693 +0000 UTC m=+1217.144155499"
Nov 02 22:18:00.156561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:00.147696    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:18:00.178556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:00.169347    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:18:00.809473 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:00.809443    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:18:00.810681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:00.810660    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:18:02.765572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:02.763715    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:18:02.771561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:02.767955    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:18:03.598985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.598954    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:18:03.604096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.604068    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.610620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.609615    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14dae2\""
Nov 02 22:18:03.615144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.615105    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:18:03.763560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.756404    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.778570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.778476    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8km2n\" (UniqueName: \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.887612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.886259    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.887612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.886422    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8km2n\" (UniqueName: \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.940595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.940560    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8km2n\" (UniqueName: \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:03.949933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:03.949899    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14dae2-84784df74c-mmptl\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") " pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:04.258669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:04.242462    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:04.918232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:04.918198    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:18:04.951466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:04.951433    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" event={"ID":"9266799c-03ef-4db5-b2f9-04dd5bdc44f1","Type":"ContainerStarted","Data":"ffff9a719db777fce367ded5025e3eb9fe2dbda7d86a2e0d3f0a2fdfee1cf6b1"}
Nov 02 22:18:06.002736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:06.002695    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" event={"ID":"9266799c-03ef-4db5-b2f9-04dd5bdc44f1","Type":"ContainerStarted","Data":"249788cc2fb9a0c2ab35c1f3313be3d0cceebce169c46c408f96812c33559283"}
Nov 02 22:18:06.023559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:06.009044    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:06.076791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:06.071797    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" podStartSLOduration=3.071766949 podStartE2EDuration="3.071766949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:06.035882286 +0000 UTC m=+1223.343973096" watchObservedRunningTime="2026-11-02 22:18:06.071766949 +0000 UTC m=+1223.379857773"
Nov 02 22:18:12.448936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:12.448881    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:18:12.449450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:12.449215    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy" containerID="cri-o://37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521" gracePeriod=30
Nov 02 22:18:17.569699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.569652    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6971-55eaf9"]
Nov 02 22:18:17.576354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.576327    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:17.589455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.589139    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-55eaf9"]
Nov 02 22:18:17.591170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.590262    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:18:17.593408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.593079    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:18:17.593408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.593187    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8p5jl\""
Nov 02 22:18:17.603749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:17.603720    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvn28\" (UniqueName: \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\") pod \"test-curl-6971-55eaf9\" (UID: \"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd\") " pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:18.067650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.018459    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvn28\" (UniqueName: \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\") pod \"test-curl-6971-55eaf9\" (UID: \"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd\") " pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:18.170146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.161889    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvn28\" (UniqueName: \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\") pod \"test-curl-6971-55eaf9\" (UID: \"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd\") " pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:18.174014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.173746    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:18:18.243176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.243141    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:18.542849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.542422    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:18:18.636147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.636115    4929 generic.go:356] "Generic (PLEG): container finished" podID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerID="37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521" exitCode=0
Nov 02 22:18:18.636641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.636233    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerDied","Data":"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"}
Nov 02 22:18:18.636641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.636277    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg" event={"ID":"50ef9959-059b-48a9-bee4-8d5e2d786e77","Type":"ContainerDied","Data":"d0d987f0964ef35a861a9a7024cc5a347028997e3ef740236a087bff721d0acf"}
Nov 02 22:18:18.636641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.636308    4929 scope.go:122] "RemoveContainer" containerID="37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"
Nov 02 22:18:18.636641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.636477    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"
Nov 02 22:18:18.652831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.652806    4929 scope.go:122] "RemoveContainer" containerID="73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"
Nov 02 22:18:18.660024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.659982    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660097    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660150    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660260    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660324    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660380    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660365    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:18.660612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660555    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660612    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660629    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:18.660668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660654    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\") pod \"50ef9959-059b-48a9-bee4-8d5e2d786e77\" (UID: \"50ef9959-059b-48a9-bee4-8d5e2d786e77\") "
Nov 02 22:18:18.660842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660822    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:18.660974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.660949    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:18.661404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661382    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:18.661770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661748    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.661868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661777    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.661868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661795    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.661868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661809    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.661868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.661823    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ef9959-059b-48a9-bee4-8d5e2d786e77-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.664035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.663004    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:18.664035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.663111    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:18:18.664035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.663381    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:18.664300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.664278    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2" pod "50ef9959-059b-48a9-bee4-8d5e2d786e77" (UID: "50ef9959-059b-48a9-bee4-8d5e2d786e77"). InnerVolumeSpecName "kube-api-access-8q9s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:18.666163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.666139    4929 scope.go:122] "RemoveContainer" containerID="37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"
Nov 02 22:18:18.666631 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:18.666606    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521\": container with ID starting with 37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521 not found: ID does not exist" containerID="37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"
Nov 02 22:18:18.666745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.666637    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521"} err="failed to get container status \"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521\": rpc error: code = NotFound desc = could not find container \"37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521\": container with ID starting with 37d53cf6dde3cf5620dd64f19d6f6ba06603b3f20c5460ae92ce20720334a521 not found: ID does not exist"
Nov 02 22:18:18.666745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.666658    4929 scope.go:122] "RemoveContainer" containerID="73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"
Nov 02 22:18:18.666998 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:18.666973    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792\": container with ID starting with 73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792 not found: ID does not exist" containerID="73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"
Nov 02 22:18:18.667093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.667006    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792"} err="failed to get container status \"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792\": rpc error: code = NotFound desc = could not find container \"73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792\": container with ID starting with 73a1be23c8827ed025e16e8f423d61258155c78538bff392a5d2d37a33788792 not found: ID does not exist"
Nov 02 22:18:18.763578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.763514    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.763578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.763567    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.763578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.763581    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ef9959-059b-48a9-bee4-8d5e2d786e77-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.763837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.763597    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9s2\" (UniqueName: \"kubernetes.io/projected/50ef9959-059b-48a9-bee4-8d5e2d786e77-kube-api-access-8q9s2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:18.825610 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:18:18.825577    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccaab18d_4b8b_4fb1_80ad_24d5ff7a69cd.slice/crio-ff9db381a170d8d32d0d6bf8db2e1efa39f00cda030eb22c62e51fe8edab0fc5 WatchSource:0}: Error finding container ff9db381a170d8d32d0d6bf8db2e1efa39f00cda030eb22c62e51fe8edab0fc5: Status 404 returned error can't find the container with id ff9db381a170d8d32d0d6bf8db2e1efa39f00cda030eb22c62e51fe8edab0fc5
Nov 02 22:18:18.825794 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.825774    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-55eaf9"]
Nov 02 22:18:18.967666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.967623    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:18:18.970236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:18.970199    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-q74cg"]
Nov 02 22:18:19.646160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:19.646124    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-55eaf9" event={"ID":"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd","Type":"ContainerStarted","Data":"8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4"}
Nov 02 22:18:19.646688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:19.646175    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-55eaf9" event={"ID":"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd","Type":"ContainerStarted","Data":"ff9db381a170d8d32d0d6bf8db2e1efa39f00cda030eb22c62e51fe8edab0fc5"}
Nov 02 22:18:19.743646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:19.723593    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6971-55eaf9" podStartSLOduration=2.7235715860000003 podStartE2EDuration="2.723571586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:19.703391749 +0000 UTC m=+1237.011482555" watchObservedRunningTime="2026-11-02 22:18:19.723571586 +0000 UTC m=+1237.031662393"
Nov 02 22:18:20.813843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:20.813809    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" path="/var/lib/kubelet/pods/50ef9959-059b-48a9-bee4-8d5e2d786e77/volumes"
Nov 02 22:18:21.485029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:21.484998    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6971-55eaf9"]
Nov 02 22:18:21.490393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:21.490360    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6971-55eaf9"]
Nov 02 22:18:21.662982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:21.662943    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6971-55eaf9" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" containerName="test-curl-6971-55eaf9" containerID="cri-o://8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4" gracePeriod=2
Nov 02 22:18:21.665333 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:21.665302    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-55eaf9\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:22.809939 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:22.809895    4929 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-55eaf9\" is forbidden: User \"system:node:ip-10-0-131-57.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.us-east-2.compute.internal' and this object" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:23.045554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.045501    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.990585    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992415    4929 cpu_manager.go:414] "containerMap: removing container" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992441    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992467    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" containerName="test-curl-6971-55eaf9"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992479    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" containerName="test-curl-6971-55eaf9"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992769    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992789    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" containerName="test-curl-6971-55eaf9"
Nov 02 22:18:23.993029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:23.992815    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:24.022034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.021920    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.030157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.021824    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:18:24.030157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.026152    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-df8835\""
Nov 02 22:18:24.130882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.123753    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:18:24.130882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.128838    4929 cpu_manager.go:414] "containerMap: removing container" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:24.130882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.129027    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ef9959-059b-48a9-bee4-8d5e2d786e77" containerName="istio-proxy"
Nov 02 22:18:24.145849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.145816    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.152090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.152053    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:18:24.173898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.172305    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-df8835-dockercfg-h7sv8\""
Nov 02 22:18:24.250148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.250013    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.250503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.250475    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfstj\" (UniqueName: \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.374822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.371014    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.374822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.371070    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfstj\" (UniqueName: \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.374822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.371103    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.374822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.371142    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7k4\" (UniqueName: \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.374822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.372414    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.387622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.386637    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfstj\" (UniqueName: \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\") pod \"payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.441585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.440196    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:24.473342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.473307    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.473569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.473346    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7k4\" (UniqueName: \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.474354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.474327    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.484354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.484325    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7k4\" (UniqueName: \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\") pod \"payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.544316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.543952    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:24.609232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.568081    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:24.706059 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.706019    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\" (UniqueName: \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\") pod \"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd\" (UID: \"ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd\") "
Nov 02 22:18:24.710578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.710502    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28" pod "ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" (UID: "ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd"). InnerVolumeSpecName "kube-api-access-kvn28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:24.812030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.811997    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvn28\" (UniqueName: \"kubernetes.io/projected/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd-kube-api-access-kvn28\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:24.814127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.814095    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" path="/var/lib/kubelet/pods/ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd/volumes"
Nov 02 22:18:24.914698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.914659    4929 generic.go:356] "Generic (PLEG): container finished" podID="ccaab18d-4b8b-4fb1-80ad-24d5ff7a69cd" containerID="8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4" exitCode=137
Nov 02 22:18:24.914884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.914705    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-55eaf9"
Nov 02 22:18:24.914884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.914754    4929 scope.go:122] "RemoveContainer" containerID="8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4"
Nov 02 22:18:24.926604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.926576    4929 scope.go:122] "RemoveContainer" containerID="8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4"
Nov 02 22:18:24.926940 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:24.926918    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4\": container with ID starting with 8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4 not found: ID does not exist" containerID="8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4"
Nov 02 22:18:24.927019 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.926959    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4"} err="failed to get container status \"8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4\": rpc error: code = NotFound desc = could not find container \"8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4\": container with ID starting with 8d1d875b37068c948d00ea961bcb144e2b49b9859dcd6f5cf69628bc29e29ca4 not found: ID does not exist"
Nov 02 22:18:24.936155 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:18:24.936113    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229f68f4_3a66_4375_b122_cfdf167fd1e9.slice/crio-61fe169ec07d7190d7c885fe3a5325996f99c91d6d3dd795e2b98ba76aac7940 WatchSource:0}: Error finding container 61fe169ec07d7190d7c885fe3a5325996f99c91d6d3dd795e2b98ba76aac7940: Status 404 returned error can't find the container with id 61fe169ec07d7190d7c885fe3a5325996f99c91d6d3dd795e2b98ba76aac7940
Nov 02 22:18:24.936589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:24.936551    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:18:25.005239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.005209    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:18:25.965566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.963032    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" event={"ID":"229f68f4-3a66-4375-b122-cfdf167fd1e9","Type":"ContainerStarted","Data":"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"}
Nov 02 22:18:25.965566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.963118    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" event={"ID":"229f68f4-3a66-4375-b122-cfdf167fd1e9","Type":"ContainerStarted","Data":"61fe169ec07d7190d7c885fe3a5325996f99c91d6d3dd795e2b98ba76aac7940"}
Nov 02 22:18:25.969568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.964913    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:25.969568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.968724    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" event={"ID":"6f36933c-5291-4a2b-9d3c-5ec3de657dd4","Type":"ContainerStarted","Data":"62fd14e52480baa51dd624e2e385936f49280bf76dc3d77f52353044ad6fa64e"}
Nov 02 22:18:25.969568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.968783    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" event={"ID":"6f36933c-5291-4a2b-9d3c-5ec3de657dd4","Type":"ContainerStarted","Data":"2956ad461df2f19738674022e65f87f984f2ca78eadf2d6bb4bc7af526e491ab"}
Nov 02 22:18:25.970360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:25.970059    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:26.007256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:26.006694    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" podStartSLOduration=3.006518198 podStartE2EDuration="3.006518198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:25.982944802 +0000 UTC m=+1243.291035612" watchObservedRunningTime="2026-11-02 22:18:26.006518198 +0000 UTC m=+1243.314609005"
Nov 02 22:18:26.080653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:26.080592    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" podStartSLOduration=3.080576753 podStartE2EDuration="3.080576753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:26.07558231 +0000 UTC m=+1243.383673118" watchObservedRunningTime="2026-11-02 22:18:26.080576753 +0000 UTC m=+1243.388667558"
Nov 02 22:18:27.499410 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:27.499374    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:18:28.045940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.045864    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" podUID="a0148602-65e7-426a-8344-3b9d162bd626" containerName="payload-processing" containerID="cri-o://8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93" gracePeriod=25
Nov 02 22:18:28.474262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.474231    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:18:28.496154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.496116    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\") pod \"a0148602-65e7-426a-8344-3b9d162bd626\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") "
Nov 02 22:18:28.496328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.496274    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\" (UniqueName: \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\") pod \"a0148602-65e7-426a-8344-3b9d162bd626\" (UID: \"a0148602-65e7-426a-8344-3b9d162bd626\") "
Nov 02 22:18:28.496526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.496499    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume" pod "a0148602-65e7-426a-8344-3b9d162bd626" (UID: "a0148602-65e7-426a-8344-3b9d162bd626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:28.497076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.497052    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0148602-65e7-426a-8344-3b9d162bd626-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:28.498593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.498524    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5" pod "a0148602-65e7-426a-8344-3b9d162bd626" (UID: "a0148602-65e7-426a-8344-3b9d162bd626"). InnerVolumeSpecName "kube-api-access-t9ls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:28.598633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:28.598553    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9ls5\" (UniqueName: \"kubernetes.io/projected/a0148602-65e7-426a-8344-3b9d162bd626-kube-api-access-t9ls5\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:29.073297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.073263    4929 generic.go:356] "Generic (PLEG): container finished" podID="a0148602-65e7-426a-8344-3b9d162bd626" containerID="8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93" exitCode=0
Nov 02 22:18:29.073503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.073323    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"
Nov 02 22:18:29.073503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.073350    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" event={"ID":"a0148602-65e7-426a-8344-3b9d162bd626","Type":"ContainerDied","Data":"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"}
Nov 02 22:18:29.073503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.073415    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n" event={"ID":"a0148602-65e7-426a-8344-3b9d162bd626","Type":"ContainerDied","Data":"396f56a70336afbbfd5514d8e090711286f20cd524919e9d1a034d1ca2d46749"}
Nov 02 22:18:29.073503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.073446    4929 scope.go:122] "RemoveContainer" containerID="8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"
Nov 02 22:18:29.084926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.084898    4929 scope.go:122] "RemoveContainer" containerID="8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"
Nov 02 22:18:29.085240 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:29.085219    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93\": container with ID starting with 8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93 not found: ID does not exist" containerID="8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"
Nov 02 22:18:29.085321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.085250    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93"} err="failed to get container status \"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93\": rpc error: code = NotFound desc = could not find container \"8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93\": container with ID starting with 8f66adf84934a33cc52a1886a8eea07a47006cbfac230f8748ce177c2436dd93 not found: ID does not exist"
Nov 02 22:18:29.096815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.096785    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:18:29.101448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:29.101422    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0dd701-5ccd64bffc-fg54n"]
Nov 02 22:18:30.808458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:30.808420    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0148602-65e7-426a-8344-3b9d162bd626" path="/var/lib/kubelet/pods/a0148602-65e7-426a-8344-3b9d162bd626/volumes"
Nov 02 22:18:31.803887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:31.803849    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:18:31.816700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:31.816667    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:18:34.026020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:34.025977    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:18:36.809978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:36.809912    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" podUID="f2997319-b8a0-464e-b791-59b4bb8537c1" containerName="payload-processing" containerID="cri-o://696d4acf16f6a563cdb267024e3b90d4b02f58731da227004aeb8ac3a5675aac" gracePeriod=25
Nov 02 22:18:36.836559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:36.817171    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" podUID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" containerName="payload-pre-processing" containerID="cri-o://51b2e42734f777eeebf6aaa7bc0056dac74a8d3723a2d995b6e20349876ed20e" gracePeriod=25
Nov 02 22:18:37.268285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.236701    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:37.373265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.345722    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:37.373265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.358066    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:37.540472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.540422    4929 generic.go:356] "Generic (PLEG): container finished" podID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" containerID="51b2e42734f777eeebf6aaa7bc0056dac74a8d3723a2d995b6e20349876ed20e" exitCode=0
Nov 02 22:18:37.540999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.540972    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" event={"ID":"f04565c1-4caa-4c23-b750-4cf05ffdcf90","Type":"ContainerDied","Data":"51b2e42734f777eeebf6aaa7bc0056dac74a8d3723a2d995b6e20349876ed20e"}
Nov 02 22:18:37.807677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.807629    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:18:37.971624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:37.971146    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:18:38.079618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.079583    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:18:38.081311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.081251    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:18:38.101992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.101950    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.103751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.103098    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.249971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.248364    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.249971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.249096    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:18:38.365560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.353643    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:18:38.365560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.354319    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:18:38.392261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.391650    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.428559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.418633    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.604625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.604584    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.810062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.763469    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:18:38.810062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.763892    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.810062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.790618    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.877620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.812917    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:38.877620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.813516    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:18:38.971587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:38.960381    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.158620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.156659    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:39.184560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.164977    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.273175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.239714    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.320711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.320658    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.373558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.348602    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:39.591558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.413339    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:39.591558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.415711    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.859839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.859791    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.986474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.902798    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:39.986474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.905429    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:18:39.986474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:39.905658    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:18:40.123385 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:40.108423    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.205s"
Nov 02 22:18:40.939695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.926704    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:18:40.960386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.952392    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.026635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.026576    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 22:18:41.026897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.275481    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.028286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.196614    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.030565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.197280    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" podUID="bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.038711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.602758    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.075691 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.342025    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.090744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.602797    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.606840    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.748624    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.749702    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.749741    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.871150    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:41.134825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.606796    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:18:41.139427 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.139379    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:18:41.168390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:40.193124    4929 generic.go:356] "Generic (PLEG): container finished" podID="f2997319-b8a0-464e-b791-59b4bb8537c1" containerID="696d4acf16f6a563cdb267024e3b90d4b02f58731da227004aeb8ac3a5675aac" exitCode=0
Nov 02 22:18:41.367679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.367630    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 02 22:18:41.513618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.391080    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\" (UniqueName: \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\") pod \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") "
Nov 02 22:18:41.513618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.441388    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\") pod \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\" (UID: \"f04565c1-4caa-4c23-b750-4cf05ffdcf90\") "
Nov 02 22:18:41.568558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.554894    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume" pod "f04565c1-4caa-4c23-b750-4cf05ffdcf90" (UID: "f04565c1-4caa-4c23-b750-4cf05ffdcf90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:41.805025 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.804904    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf" pod "f04565c1-4caa-4c23-b750-4cf05ffdcf90" (UID: "f04565c1-4caa-4c23-b750-4cf05ffdcf90"). InnerVolumeSpecName "kube-api-access-zg9mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:41.811649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.811602    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9mf\" (UniqueName: \"kubernetes.io/projected/f04565c1-4caa-4c23-b750-4cf05ffdcf90-kube-api-access-zg9mf\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:41.813528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.813093    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04565c1-4caa-4c23-b750-4cf05ffdcf90-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:41.876179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.876134    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" event={"ID":"f2997319-b8a0-464e-b791-59b4bb8537c1","Type":"ContainerDied","Data":"696d4acf16f6a563cdb267024e3b90d4b02f58731da227004aeb8ac3a5675aac"}
Nov 02 22:18:41.876514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.876494    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:18:41.876954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.876934    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:18:41.877142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.877128    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:18:41.891565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.877651    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:18:41.912376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.912347    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:18:41.912776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.912744    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:18:41.913284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.913267    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:18:41.913565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.913553    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:18:41.913760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.913750    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:18:41.913984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.913974    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:18:41.914154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:41.914145    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:18:42.045186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.045136    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.047100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.047056    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.047504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.047479    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.087441 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.087339    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.087837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.087808    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.136102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.136051    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:42.154557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.137684    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.155240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.155195    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:42.183557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.158464    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:42.498562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.202721    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.498562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.202784    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.498562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.202807    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 22:18:42.498562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.218198    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:42.548082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.548048    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:18:42.575686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:42.575638    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:18:43.299275 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.299198    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:43.503589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.490626    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"
Nov 02 22:18:43.524437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.524400    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:18:43.603334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.524585    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:18:43.707962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.524746    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:43.731315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.728934    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt" event={"ID":"f04565c1-4caa-4c23-b750-4cf05ffdcf90","Type":"ContainerDied","Data":"e3dedbe81fe8c074d5072b1cf03b0f6653d60653e723789111fd3e9aff337582"}
Nov 02 22:18:43.731315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.729034    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:18:43.731315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.729057    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:18:43.731315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.729262    4929 scope.go:122] "RemoveContainer" containerID="51b2e42734f777eeebf6aaa7bc0056dac74a8d3723a2d995b6e20349876ed20e"
Nov 02 22:18:43.762558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.759075    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787411    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688" event={"ID":"f2997319-b8a0-464e-b791-59b4bb8537c1","Type":"ContainerDied","Data":"9347c2e412b9cd5cae750285a5f8121d46f9a102f7288b0291e63a68d7ede366"}
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787564    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787611    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787644    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787675    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787705    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787736    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787769    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787799    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787831    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:18:43.797561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.787862    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:18:43.809434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.807517    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\" (UniqueName: \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\") pod \"f2997319-b8a0-464e-b791-59b4bb8537c1\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") "
Nov 02 22:18:43.809824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.809800    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\") pod \"f2997319-b8a0-464e-b791-59b4bb8537c1\" (UID: \"f2997319-b8a0-464e-b791-59b4bb8537c1\") "
Nov 02 22:18:43.810415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.810301    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume" pod "f2997319-b8a0-464e-b791-59b4bb8537c1" (UID: "f2997319-b8a0-464e-b791-59b4bb8537c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:43.813685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.813317    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:18:43.818488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.818436    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:18:43.826664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.826183    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h" pod "f2997319-b8a0-464e-b791-59b4bb8537c1" (UID: "f2997319-b8a0-464e-b791-59b4bb8537c1"). InnerVolumeSpecName "kube-api-access-rqb2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:43.826664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.826518    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2997319-b8a0-464e-b791-59b4bb8537c1-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:43.893434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.893031    4929 scope.go:122] "RemoveContainer" containerID="696d4acf16f6a563cdb267024e3b90d4b02f58731da227004aeb8ac3a5675aac"
Nov 02 22:18:43.928578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:43.928528    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb2h\" (UniqueName: \"kubernetes.io/projected/f2997319-b8a0-464e-b791-59b4bb8537c1-kube-api-access-rqb2h\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:45.172845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:45.125875    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:18:45.172845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:45.125929    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84a94060-6649f69d59-249xt"]
Nov 02 22:18:46.362578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.360603    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:18:46.365527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.365497    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84a94060-6c8f769dbc-9x688"]
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.374409    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376856    4929 cpu_manager.go:414] "containerMap: removing container" podUID="f2997319-b8a0-464e-b791-59b4bb8537c1" containerName="payload-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376878    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2997319-b8a0-464e-b791-59b4bb8537c1" containerName="payload-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376926    4929 cpu_manager.go:414] "containerMap: removing container" podUID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" containerName="payload-pre-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376934    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" containerName="payload-pre-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376979    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a0148602-65e7-426a-8344-3b9d162bd626" containerName="payload-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.376988    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0148602-65e7-426a-8344-3b9d162bd626" containerName="payload-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.377209    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2997319-b8a0-464e-b791-59b4bb8537c1" containerName="payload-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.377230    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" containerName="payload-pre-processing"
Nov 02 22:18:46.379839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.377247    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0148602-65e7-426a-8344-3b9d162bd626" containerName="payload-processing"
Nov 02 22:18:46.396810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.391629    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.398829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.398807    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e30968-openshift-default-dockercfg-vjmjz\""
Nov 02 22:18:46.406013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.405939    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:18:46.462945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.462896    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.463988 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.463960    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464017    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464127    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464242    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464302 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464272    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d9v\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464517    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464580    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.464689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.464605    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571034    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571164    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571214    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571292    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571420    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571472    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571568    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571690    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.571723    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d9v\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.572767    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.580804    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.584693    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.584990    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.586413    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.597709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.586714    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.617566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.615880    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.620566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.618228    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.620566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.618489    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d9v\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\") pod \"e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") " pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.757559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.756781    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:46.903565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.902579    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04565c1-4caa-4c23-b750-4cf05ffdcf90" path="/var/lib/kubelet/pods/f04565c1-4caa-4c23-b750-4cf05ffdcf90/volumes"
Nov 02 22:18:46.903565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.903440    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2997319-b8a0-464e-b791-59b4bb8537c1" path="/var/lib/kubelet/pods/f2997319-b8a0-464e-b791-59b4bb8537c1/volumes"
Nov 02 22:18:46.942557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:46.942330    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerName="payload-processing" containerID="cri-o://073aa473221ef348f094f7aaff740585ff315b09773bc06eea8d1b8398f6beae" gracePeriod=25
Nov 02 22:18:47.199149 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.199118    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:18:47.284376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.283637    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:47.284376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.283783    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:47.284376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.283830    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:47.685790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.685691    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerStarted","Data":"99791ee724799d87023270b48c00253aaeb4b8139ea3ddbffd92b1e184e8e987"}
Nov 02 22:18:47.688332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.688099    4929 generic.go:356] "Generic (PLEG): container finished" podID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerID="073aa473221ef348f094f7aaff740585ff315b09773bc06eea8d1b8398f6beae" exitCode=0
Nov 02 22:18:47.688332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.688202    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" event={"ID":"bb1fa85d-e0bb-4609-b720-4ca63d8e5460","Type":"ContainerDied","Data":"073aa473221ef348f094f7aaff740585ff315b09773bc06eea8d1b8398f6beae"}
Nov 02 22:18:47.688332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.688247    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" event={"ID":"bb1fa85d-e0bb-4609-b720-4ca63d8e5460","Type":"ContainerDied","Data":"f2886a26327df592ed2739b2760d7f1c129685f44588296bd59a62d2f70df40d"}
Nov 02 22:18:47.688332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.688271    4929 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2886a26327df592ed2739b2760d7f1c129685f44588296bd59a62d2f70df40d"
Nov 02 22:18:47.690755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.689913    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:18:47.809104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.809017    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\") pod \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") "
Nov 02 22:18:47.823556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.809356    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\" (UniqueName: \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\") pod \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\" (UID: \"bb1fa85d-e0bb-4609-b720-4ca63d8e5460\") "
Nov 02 22:18:47.856135 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.810471    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume" pod "bb1fa85d-e0bb-4609-b720-4ca63d8e5460" (UID: "bb1fa85d-e0bb-4609-b720-4ca63d8e5460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:47.917839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.917780    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m" pod "bb1fa85d-e0bb-4609-b720-4ca63d8e5460" (UID: "bb1fa85d-e0bb-4609-b720-4ca63d8e5460"). InnerVolumeSpecName "kube-api-access-cjm4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:47.959715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.959683    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:47.959917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:47.959899    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjm4m\" (UniqueName: \"kubernetes.io/projected/bb1fa85d-e0bb-4609-b720-4ca63d8e5460-kube-api-access-cjm4m\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:48.428416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.428370    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.123:50052\" within 1s: context deadline exceeded"
Nov 02 22:18:48.429050 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:18:48.428804    4929 logging.go:55] [core] [Channel #1994 SubChannel #1995]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.123:50052", ServerName: "10.133.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.123:50052: operation was canceled"
Nov 02 22:18:48.700877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.700773    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerStarted","Data":"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"}
Nov 02 22:18:48.700877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.700866    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"
Nov 02 22:18:48.729471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.729410    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podStartSLOduration=2.729390413 podStartE2EDuration="2.729390413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:48.721263167 +0000 UTC m=+1266.029353973" watchObservedRunningTime="2026-11-02 22:18:48.729390413 +0000 UTC m=+1266.037481219"
Nov 02 22:18:48.730121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.730090    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" podUID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" containerName="payload-pre-processing" containerID="cri-o://0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04" gracePeriod=25
Nov 02 22:18:48.747110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.747081    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:18:48.751610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.751581    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d9416f-8bb4d5c4c-mz9ft"]
Nov 02 22:18:48.757832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.757808    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:48.763994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.763968    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:48.810598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:48.810521    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" path="/var/lib/kubelet/pods/bb1fa85d-e0bb-4609-b720-4ca63d8e5460/volumes"
Nov 02 22:18:49.357892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.357861    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:18:49.398088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.397994    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\" (UniqueName: \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\") pod \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") "
Nov 02 22:18:49.398783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.398759    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\") pod \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\" (UID: \"3fcc68a1-888c-4dc2-887f-acbbb52cb850\") "
Nov 02 22:18:49.399605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.399236    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume" pod "3fcc68a1-888c-4dc2-887f-acbbb52cb850" (UID: "3fcc68a1-888c-4dc2-887f-acbbb52cb850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:49.400433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.400410    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcc68a1-888c-4dc2-887f-acbbb52cb850-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:49.408990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.405810    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc" pod "3fcc68a1-888c-4dc2-887f-acbbb52cb850" (UID: "3fcc68a1-888c-4dc2-887f-acbbb52cb850"). InnerVolumeSpecName "kube-api-access-g6khc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:49.502417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.502378    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6khc\" (UniqueName: \"kubernetes.io/projected/3fcc68a1-888c-4dc2-887f-acbbb52cb850-kube-api-access-g6khc\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:49.712837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.712017    4929 generic.go:356] "Generic (PLEG): container finished" podID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" containerID="0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04" exitCode=0
Nov 02 22:18:49.714607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.712359    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"
Nov 02 22:18:49.714866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.714836    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" event={"ID":"3fcc68a1-888c-4dc2-887f-acbbb52cb850","Type":"ContainerDied","Data":"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"}
Nov 02 22:18:49.720903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.715077    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86" event={"ID":"3fcc68a1-888c-4dc2-887f-acbbb52cb850","Type":"ContainerDied","Data":"40d72038ba0de9c13b83a268cc424e8e47d82aaf475cf8b033974563ecfc0c52"}
Nov 02 22:18:49.720903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.715230    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:49.720903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.715654    4929 scope.go:122] "RemoveContainer" containerID="0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"
Nov 02 22:18:49.720903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.717169    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:18:49.763151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.748347    4929 scope.go:122] "RemoveContainer" containerID="0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"
Nov 02 22:18:49.763151 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:49.749988    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04\": container with ID starting with 0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04 not found: ID does not exist" containerID="0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"
Nov 02 22:18:49.763151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.750033    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04"} err="failed to get container status \"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04\": rpc error: code = NotFound desc = could not find container \"0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04\": container with ID starting with 0bfa031130e99b95ba0eed0d7584dc9d332c6edc4c2da74557a62b4fdff29d04 not found: ID does not exist"
Nov 02 22:18:49.996632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:49.996285    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:18:50.003080 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:50.000972    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d9416f-756d6bc9f-7cg86"]
Nov 02 22:18:50.809296 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:50.809264    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" path="/var/lib/kubelet/pods/3fcc68a1-888c-4dc2-887f-acbbb52cb850/volumes"
Nov 02 22:18:50.975836 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:50.975801    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.441505    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.442832    4929 cpu_manager.go:414] "containerMap: removing container" podUID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" containerName="payload-pre-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.442857    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" containerName="payload-pre-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.442882    4929 cpu_manager.go:414] "containerMap: removing container" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerName="payload-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.442890    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerName="payload-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.443126    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fcc68a1-888c-4dc2-887f-acbbb52cb850" containerName="payload-pre-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.443143    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1fa85d-e0bb-4609-b720-4ca63d8e5460" containerName="payload-processing"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.459978    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.460065    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.526148    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e30968-dockercfg-2wds5\""
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.526245    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e30968\""
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.563033    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.563134    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswf2\" (UniqueName: \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.772420    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.780080    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.772545    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswf2\" (UniqueName: \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:52.865706    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswf2\" (UniqueName: \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\") pod \"payload-processing-e2e-inf-a-e30968-749646c859-88gln\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.090956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.089199    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:53.171280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.171244    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:18:53.179529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.179069    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.202714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.202606    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:18:53.202926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.202905    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7bb56e-openshift-default-dockercfg-jkclr\""
Nov 02 22:18:53.237958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.237929    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238107    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238149    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238331 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238302    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7qz\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238352    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238464    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238505    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238601    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.238731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.238644    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379492    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379728    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379790    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379854    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7qz\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379887    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.379975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379939    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.380139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.379975    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.380139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.380030    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.380139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.380079    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.411562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.408098    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.411562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.408415    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.411562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.408990    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.411562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.409231    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.434564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.432818    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.438630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.438596    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.481554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.450193    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.481554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.450715    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7qz\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.481554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.479941    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.531676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.531636    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:18:53.575560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.568956    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:53.866092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.866054    4929 generic.go:356] "Generic (PLEG): container finished" podID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerID="d42dac41d7c4a660ebc0db959be99516219fc984ea974b74f4b96cd006b89221" exitCode=0
Nov 02 22:18:53.866412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.866206    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerDied","Data":"d42dac41d7c4a660ebc0db959be99516219fc984ea974b74f4b96cd006b89221"}
Nov 02 22:18:53.867234 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.867210    4929 scope.go:122] "RemoveContainer" containerID="d42dac41d7c4a660ebc0db959be99516219fc984ea974b74f4b96cd006b89221"
Nov 02 22:18:53.871074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.871039    4929 generic.go:356] "Generic (PLEG): container finished" podID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerID="bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab" exitCode=0
Nov 02 22:18:53.871189 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.871175    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerDied","Data":"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"}
Nov 02 22:18:53.872115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.871800    4929 scope.go:122] "RemoveContainer" containerID="bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"
Nov 02 22:18:53.877445 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:53.874321    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:18:53.879896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.879866    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" event={"ID":"616c0e90-878a-4090-8504-ecaea0c5d1d0","Type":"ContainerStarted","Data":"4efe25a72e7433a1b3ed8f74f651be08c984cc001e80aa1a44f45714801a1941"}
Nov 02 22:18:53.881311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881276    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:53.881414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881337    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:53.881490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881447    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:53.881490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881459    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:53.881663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881567    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:53.881663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:53.881607    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:54.263868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.262835    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:18:54.284973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.284926    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:54.286866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.286832    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:54.287083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.287048    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:18:54.904651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.904609    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerStarted","Data":"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"}
Nov 02 22:18:54.904863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.904672    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerStarted","Data":"8f2c032897dbee0350ff350d4eff113bcbfaa47a8d8943f1ade80e0abcf54bfe"}
Nov 02 22:18:54.907116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.907084    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerStarted","Data":"445e839b1cf592918352614c41549bfc2ef9a945392d8e333907103434a52c13"}
Nov 02 22:18:54.910219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.909492    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerStarted","Data":"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"}
Nov 02 22:18:54.911329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.911306    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" event={"ID":"616c0e90-878a-4090-8504-ecaea0c5d1d0","Type":"ContainerStarted","Data":"03b86023f7a6b25cda481d57e23d337ae83926a5edce978987331711d712a57d"}
Nov 02 22:18:54.911490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.911472    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:18:54.952573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:54.934438    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podStartSLOduration=1.934415218 podStartE2EDuration="1.934415218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:54.92391356 +0000 UTC m=+1272.232004364" watchObservedRunningTime="2026-11-02 22:18:54.934415218 +0000 UTC m=+1272.242506024"
Nov 02 22:18:55.086560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.084349    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" podStartSLOduration=3.084327621 podStartE2EDuration="3.084327621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:54.979011794 +0000 UTC m=+1272.287102599" watchObservedRunningTime="2026-11-02 22:18:55.084327621 +0000 UTC m=+1272.392418429"
Nov 02 22:18:55.158996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.158839    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:18:55.189026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.188675    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:18:55.189026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.188737    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:18:55.451659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.449210    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:18:55.452718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.452685    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:18:55.511879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.507145    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:18:55.601598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.585224    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:55.601598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:55.597308    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:56.003084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.002516    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:56.022158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.022125    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:18:56.220342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.220301    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:18:56.220688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.220640    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:18:56.427509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.427472    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:18:56.427696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:56.427526    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:18:57.167561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:57.157565    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:18:57.167561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:57.157642    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:18:57.428202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:57.428108    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:18:57.428202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:57.428164    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:18:57.520336 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:18:57.520298    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:18:57.974556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:57.974075    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" containerID="cri-o://43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e" gracePeriod=45
Nov 02 22:18:58.014238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.014195    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": EOF"
Nov 02 22:18:58.167559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.152993    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:18:58.167559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.153059    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:18:58.429266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.429209    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:18:58.429564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.429288    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:18:58.532194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.532153    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:18:58.605863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.605820    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:18:58.606068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.605956    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.607927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.607900    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:18:58.609016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.608990    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.609118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.609099    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfrz\" (UniqueName: \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.613610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.613396    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-7bb56e\""
Nov 02 22:18:58.613610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.613417    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-7bb56e-dockercfg-jxj8m\""
Nov 02 22:18:58.614985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.614961    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:58.618897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.618867    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:18:58.627466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.627128    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:18:58.727617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.711111    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.727617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.711320    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfrz\" (UniqueName: \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.727617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.715694    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:58.877734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.876874    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.877734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.877040    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.877734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.877089    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.893557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.892669    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.893557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.892749    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\" (UniqueName: \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.893557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.892895    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\") pod \"06984497-c159-4407-8bdd-10db029492ed\" (UID: \"06984497-c159-4407-8bdd-10db029492ed\") "
Nov 02 22:18:58.893876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.893791    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tg2\" (UniqueName: \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.050417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.893962    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.050417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.929808    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:59.050417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:58.942575    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:59.100555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.075071    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tg2\" (UniqueName: \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.100555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.075233    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.100555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.075518    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.100555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.075551    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.100555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.077243    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.115197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.114459    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tg2\" (UniqueName: \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\") pod \"payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.115197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.114586    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:59.198562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.159370    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:18:59.383138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.383079    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:18:59.388702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.380009    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:18:59.389455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.389424    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:59.489613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.467593    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.566560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.479189    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfrz\" (UniqueName: \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\") pod \"payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:59.566560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.512150    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz" pod "06984497-c159-4407-8bdd-10db029492ed" (UID: "06984497-c159-4407-8bdd-10db029492ed"). InnerVolumeSpecName "kube-api-access-lptxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:59.627562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.603315    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:18:59.695609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.695154    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:18:59.771570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.761002    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:59.820560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.795329    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06984497-c159-4407-8bdd-10db029492ed-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.820910 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.820881    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lptxz\" (UniqueName: \"kubernetes.io/projected/06984497-c159-4407-8bdd-10db029492ed-kube-api-access-lptxz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.821052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.821035    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06984497-c159-4407-8bdd-10db029492ed-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:59.885528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.885476    4929 generic.go:356] "Generic (PLEG): container finished" podID="06984497-c159-4407-8bdd-10db029492ed" containerID="43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e" exitCode=0
Nov 02 22:18:59.886160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.886116    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" event={"ID":"06984497-c159-4407-8bdd-10db029492ed","Type":"ContainerDied","Data":"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"}
Nov 02 22:18:59.886378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.886347    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f" event={"ID":"06984497-c159-4407-8bdd-10db029492ed","Type":"ContainerDied","Data":"95805c86c292cf1108aa824c2fe780278e406806c7e48cd5fe490020174f171c"}
Nov 02 22:18:59.886851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.886820    4929 scope.go:122] "RemoveContainer" containerID="43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"
Nov 02 22:18:59.905217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.905184    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"
Nov 02 22:18:59.908776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.908744    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:18:59.941919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.941860    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:18:59.942754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.942711    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:18:59.945156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.941727    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:18:59.946463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.944147    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:18:59.946663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:18:59.946637    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:19:00.136229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.136194    4929 scope.go:122] "RemoveContainer" containerID="43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"
Nov 02 22:19:00.137088 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:00.137049    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e\": container with ID starting with 43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e not found: ID does not exist" containerID="43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"
Nov 02 22:19:00.137313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.137280    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e"} err="failed to get container status \"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e\": rpc error: code = NotFound desc = could not find container \"43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e\": container with ID starting with 43fdf78b079425a890d702ed2879e33bf6b80d7026797a5dcc404b618e73055e not found: ID does not exist"
Nov 02 22:19:00.171359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.171315    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:19:00.171650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.171612    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:19:00.178101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.178054    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:00.415728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.411865    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:19:00.422799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.419494    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-84a94060/e2e-lifecycle-model-84a94060-kserve-7cbcc74d9c-z267f"]
Nov 02 22:19:00.428494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.428433    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:19:00.428652 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.428508    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:19:00.976193 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:00.919579    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:19:01.014162 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:01.014061    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe82fb9_ba5d_41f2_ad57_b14d3828f3cc.slice/crio-bbe7c467f2ad18f28a473a76c330ecd7b2ed45d546d36e2191f73f07a3b81c01 WatchSource:0}: Error finding container bbe7c467f2ad18f28a473a76c330ecd7b2ed45d546d36e2191f73f07a3b81c01: Status 404 returned error can't find the container with id bbe7c467f2ad18f28a473a76c330ecd7b2ed45d546d36e2191f73f07a3b81c01
Nov 02 22:19:01.023725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.023694    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06984497-c159-4407-8bdd-10db029492ed" path="/var/lib/kubelet/pods/06984497-c159-4407-8bdd-10db029492ed/volumes"
Nov 02 22:19:01.024229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.024204    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:19:01.092915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.092880    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:19:01.097746 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:01.097709    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39465d60_8f30_436d_b5df_e7d25adbca88.slice/crio-895cf1ee17ee81734887d36c03ff5ec18ae064194c5354ec916f313fec8d0ede WatchSource:0}: Error finding container 895cf1ee17ee81734887d36c03ff5ec18ae064194c5354ec916f313fec8d0ede: Status 404 returned error can't find the container with id 895cf1ee17ee81734887d36c03ff5ec18ae064194c5354ec916f313fec8d0ede
Nov 02 22:19:01.156558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.151149    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 22:19:01.156558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.151228    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 22:19:01.455927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.455832    4929 patch_prober.go:29] interesting pod/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 02 22:19:01.490837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:01.490731    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 02 22:19:02.143485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.143029    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" event={"ID":"39465d60-8f30-436d-b5df-e7d25adbca88","Type":"ContainerStarted","Data":"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"}
Nov 02 22:19:02.143485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.143100    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" event={"ID":"39465d60-8f30-436d-b5df-e7d25adbca88","Type":"ContainerStarted","Data":"895cf1ee17ee81734887d36c03ff5ec18ae064194c5354ec916f313fec8d0ede"}
Nov 02 22:19:02.143485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.143155    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:19:02.148100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.145966    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" event={"ID":"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc","Type":"ContainerStarted","Data":"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"}
Nov 02 22:19:02.148100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.146028    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" event={"ID":"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc","Type":"ContainerStarted","Data":"bbe7c467f2ad18f28a473a76c330ecd7b2ed45d546d36e2191f73f07a3b81c01"}
Nov 02 22:19:02.148100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.147129    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:19:02.174985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.174747    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:19:02.178277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.178224    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:19:02.180607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.180553    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" podStartSLOduration=4.180521662 podStartE2EDuration="4.180521662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:02.160698644 +0000 UTC m=+1279.468789453" watchObservedRunningTime="2026-11-02 22:19:02.180521662 +0000 UTC m=+1279.488612481"
Nov 02 22:19:02.191090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.191035    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" podStartSLOduration=4.191018837 podStartE2EDuration="4.191018837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:02.188761472 +0000 UTC m=+1279.496852284" watchObservedRunningTime="2026-11-02 22:19:02.191018837 +0000 UTC m=+1279.499109641"
Nov 02 22:19:02.491692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.489962    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:19:02.492916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:02.492890    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:19:03.988308 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:03.988267    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:05.910557 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:05.904028    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:06.006203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:06.006171    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:19:06.721245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:06.721204    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:19:06.721496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:06.721466    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy" containerID="cri-o://495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa" gracePeriod=30
Nov 02 22:19:12.160333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.160305    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:19:12.245795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.245753    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.245995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.245819    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.245995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.245850    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.245995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.245934    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246022    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246099    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246113    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:12.246299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246285    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246346    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:12.246458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246439    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246610    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:12.246724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246706    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\") pod \"72fa532d-5c35-49ee-97b1-7b21d30956b3\" (UID: \"72fa532d-5c35-49ee-97b1-7b21d30956b3\") "
Nov 02 22:19:12.246870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.246844    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:12.247310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.247287    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:12.248259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248227    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.248370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248266    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.248370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248281    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fa532d-5c35-49ee-97b1-7b21d30956b3-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.248370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248295    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.248370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248309    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.248737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.248705    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:19:12.249227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.249202    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "kube-api-access-dl64m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:12.249421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.249389    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:12.249829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.249802    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token" pod "72fa532d-5c35-49ee-97b1-7b21d30956b3" (UID: "72fa532d-5c35-49ee-97b1-7b21d30956b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:12.348993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.348897    4929 generic.go:356] "Generic (PLEG): container finished" podID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerID="495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa" exitCode=0
Nov 02 22:19:12.348993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.348951    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerDied","Data":"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"}
Nov 02 22:19:12.348993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.348977    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"
Nov 02 22:19:12.349248 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349036    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p" event={"ID":"72fa532d-5c35-49ee-97b1-7b21d30956b3","Type":"ContainerDied","Data":"44c95a3b1e6110508cc382d73ca9b402e234f1c102ba7224f1805da40b5d8957"}
Nov 02 22:19:12.349248 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349059    4929 scope.go:122] "RemoveContainer" containerID="495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"
Nov 02 22:19:12.349892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349867    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.349892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349894    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.350050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349905    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl64m\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-kube-api-access-dl64m\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.350050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.349918    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fa532d-5c35-49ee-97b1-7b21d30956b3-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:12.361784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.361764    4929 scope.go:122] "RemoveContainer" containerID="bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"
Nov 02 22:19:12.376265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.376234    4929 scope.go:122] "RemoveContainer" containerID="495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"
Nov 02 22:19:12.376715 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:12.376689    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa\": container with ID starting with 495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa not found: ID does not exist" containerID="495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"
Nov 02 22:19:12.376832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.376721    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa"} err="failed to get container status \"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa\": rpc error: code = NotFound desc = could not find container \"495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa\": container with ID starting with 495672ac7af948b0860f01ce7d1b6983bcca8eee9dd0a5f9c2952045cff534fa not found: ID does not exist"
Nov 02 22:19:12.376832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.376740    4929 scope.go:122] "RemoveContainer" containerID="bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"
Nov 02 22:19:12.377474 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:12.377452    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab\": container with ID starting with bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab not found: ID does not exist" containerID="bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"
Nov 02 22:19:12.377612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.377479    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab"} err="failed to get container status \"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab\": rpc error: code = NotFound desc = could not find container \"bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab\": container with ID starting with bf5ee8441e9d0cda79e9d361800f03cc9234b70bc174e8e9d9bc49a7751b21ab not found: ID does not exist"
Nov 02 22:19:12.379085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.379067    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:19:12.383624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.383603    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d9416f-openshift-default-5fcbc8fc89-rc27p"]
Nov 02 22:19:12.807682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:12.807648    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" path="/var/lib/kubelet/pods/72fa532d-5c35-49ee-97b1-7b21d30956b3/volumes"
Nov 02 22:19:13.239557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:13.231748    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:19:14.094194 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:14.094160    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:14.202061 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:14.202026    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:19:15.404137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:15.404097    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:19:15.635571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:15.635424    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:19:15.672626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:15.666206    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" containerID="cri-o://ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad" gracePeriod=30
Nov 02 22:19:17.434338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.434279    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:19:17.441675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.441640    4929 cpu_manager.go:414] "containerMap: removing container" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:17.443279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.443255    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:17.443586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.443570    4929 cpu_manager.go:414] "containerMap: removing container" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main"
Nov 02 22:19:17.443695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.443682    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main"
Nov 02 22:19:17.444596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.444349    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="06984497-c159-4407-8bdd-10db029492ed" containerName="main"
Nov 02 22:19:17.445012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.444996    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:17.445170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.445158    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:17.453150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.453116    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.456253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.455688    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:19:17.471584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.465152    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-00c72a-openshift-default-dockercfg-pdxkr\""
Nov 02 22:19:17.577517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.577486    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:19:17.664833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.664795    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.664958    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665023    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665055    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665155    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665193    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665269    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665326    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzgn\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.665478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.665427    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.775800    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.775887    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.775968    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776043    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzgn\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776108    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776203    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776301    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776345    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.776379    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.782607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.777983    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.788557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.784075    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.788557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.784340    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.788557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.784592    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.794558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.790065    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.798877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.798638    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.818554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.813814    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.821469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.821438    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.853571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.849868    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzgn\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:17.898904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:17.898865    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:18.808184 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:18.808149    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a67bb9_b21b_4a9f_ad9a_73704cae7dab.slice/crio-beddc2aa941a543e1ab3485e99430b1f3fd9a596432dadc5729be3590392700e WatchSource:0}: Error finding container beddc2aa941a543e1ab3485e99430b1f3fd9a596432dadc5729be3590392700e: Status 404 returned error can't find the container with id beddc2aa941a543e1ab3485e99430b1f3fd9a596432dadc5729be3590392700e
Nov 02 22:19:18.811269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:18.811241    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:19:18.811771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:18.811737    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:19:18.811829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:18.811810    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:19:18.811865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:18.811853    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:19:19.485023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.484986    4929 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:19:19.485207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.485047    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:19:19.751105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.750983    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerStarted","Data":"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"}
Nov 02 22:19:19.751105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.751055    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerStarted","Data":"beddc2aa941a543e1ab3485e99430b1f3fd9a596432dadc5729be3590392700e"}
Nov 02 22:19:19.773165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.773110    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podStartSLOduration=2.773095357 podStartE2EDuration="2.773095357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:19.770463076 +0000 UTC m=+1297.078553884" watchObservedRunningTime="2026-11-02 22:19:19.773095357 +0000 UTC m=+1297.081186162"
Nov 02 22:19:19.900067 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.900033    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:19.905650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:19.905618    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:20.905903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:20.905577    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:20.905903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:20.905706    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:19:21.028127 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:21.028092    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:21.638557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.634157    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:19:21.710909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.710874    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.710969    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711091    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711307    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:21.711444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711420    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:21.711489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711438    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:21.711583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711520    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711660 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711594    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711736    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711827    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.711909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.711888    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.712338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.712311    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:21.712464 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.712415    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\") pod \"8cbc4305-7123-41ad-aea4-b34b96d69fec\" (UID: \"8cbc4305-7123-41ad-aea4-b34b96d69fec\") "
Nov 02 22:19:21.712711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.712665    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:21.713670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.713646    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.713855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.713671    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.714996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.714967    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbc4305-7123-41ad-aea4-b34b96d69fec-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.715110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.715002    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.715110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.715017    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.719928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.719883    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:21.719928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.719892    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:19:21.723630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.720738    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:21.724148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.724122    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx" pod "8cbc4305-7123-41ad-aea4-b34b96d69fec" (UID: "8cbc4305-7123-41ad-aea4-b34b96d69fec"). InnerVolumeSpecName "kube-api-access-q4jfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:21.816708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.816610    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4jfx\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-kube-api-access-q4jfx\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.816708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.816646    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.816708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.816662    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.816708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.816676    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbc4305-7123-41ad-aea4-b34b96d69fec-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:21.882176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.882144    4929 generic.go:356] "Generic (PLEG): container finished" podID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerID="ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad" exitCode=0
Nov 02 22:19:21.882367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.882191    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"
Nov 02 22:19:21.882367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.882220    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" event={"ID":"8cbc4305-7123-41ad-aea4-b34b96d69fec","Type":"ContainerDied","Data":"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"}
Nov 02 22:19:21.882367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.882274    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b" event={"ID":"8cbc4305-7123-41ad-aea4-b34b96d69fec","Type":"ContainerDied","Data":"3d62c1e81f025be73cb7dc08dfaf434e21ebef78878dc75f218ad2c8c13dc119"}
Nov 02 22:19:21.882367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.882296    4929 scope.go:122] "RemoveContainer" containerID="ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"
Nov 02 22:19:21.894476 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.894455    4929 scope.go:122] "RemoveContainer" containerID="ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"
Nov 02 22:19:21.894822 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:21.894797    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad\": container with ID starting with ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad not found: ID does not exist" containerID="ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"
Nov 02 22:19:21.894880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.894835    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad"} err="failed to get container status \"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad\": rpc error: code = NotFound desc = could not find container \"ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad\": container with ID starting with ed04f6efd19d376abaa10a23ad0437f8777dfca3871e29cea390d7290b16faad not found: ID does not exist"
Nov 02 22:19:21.909352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.909328    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:19:21.915606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:21.915580    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-n2s2b"]
Nov 02 22:19:22.601557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:22.597989    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" podUID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" containerName="payload-processing" containerID="cri-o://249788cc2fb9a0c2ab35c1f3313be3d0cceebce169c46c408f96812c33559283" gracePeriod=25
Nov 02 22:19:22.822757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:22.822719    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" path="/var/lib/kubelet/pods/8cbc4305-7123-41ad-aea4-b34b96d69fec/volumes"
Nov 02 22:19:22.900709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:22.900675    4929 generic.go:356] "Generic (PLEG): container finished" podID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" containerID="249788cc2fb9a0c2ab35c1f3313be3d0cceebce169c46c408f96812c33559283" exitCode=0
Nov 02 22:19:22.900823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:22.900782    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" event={"ID":"9266799c-03ef-4db5-b2f9-04dd5bdc44f1","Type":"ContainerDied","Data":"249788cc2fb9a0c2ab35c1f3313be3d0cceebce169c46c408f96812c33559283"}
Nov 02 22:19:22.932870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:22.932845    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:19:23.040980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.040909    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\") pod \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") "
Nov 02 22:19:23.041196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.041154    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\" (UniqueName: \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\") pod \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\" (UID: \"9266799c-03ef-4db5-b2f9-04dd5bdc44f1\") "
Nov 02 22:19:23.041448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.041419    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume" pod "9266799c-03ef-4db5-b2f9-04dd5bdc44f1" (UID: "9266799c-03ef-4db5-b2f9-04dd5bdc44f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:23.042215 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.042188    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:23.055582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.048927    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n" pod "9266799c-03ef-4db5-b2f9-04dd5bdc44f1" (UID: "9266799c-03ef-4db5-b2f9-04dd5bdc44f1"). InnerVolumeSpecName "kube-api-access-8km2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:23.143472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.143381    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8km2n\" (UniqueName: \"kubernetes.io/projected/9266799c-03ef-4db5-b2f9-04dd5bdc44f1-kube-api-access-8km2n\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:23.274723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.274684    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:19:23.275624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275603    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" containerName="payload-processing"
Nov 02 22:19:23.275624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275622    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" containerName="payload-processing"
Nov 02 22:19:23.275785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275642    4929 cpu_manager.go:414] "containerMap: removing container" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy"
Nov 02 22:19:23.275785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275649    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy"
Nov 02 22:19:23.275785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275663    4929 cpu_manager.go:414] "containerMap: removing container" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:23.275785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275668    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fa532d-5c35-49ee-97b1-7b21d30956b3" containerName="istio-proxy"
Nov 02 22:19:23.275914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275796    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" containerName="payload-processing"
Nov 02 22:19:23.275914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.275808    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbc4305-7123-41ad-aea4-b34b96d69fec" containerName="istio-proxy"
Nov 02 22:19:23.279789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.279768    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.285728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.285001    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-00c72a\""
Nov 02 22:19:23.290409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.290383    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:19:23.309131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.309106    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:19:23.313882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.313855    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.322378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.322095    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-00c72a-dockercfg-8rrv5\""
Nov 02 22:19:23.326574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.325310    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:19:23.349779 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.349739    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88jq\" (UniqueName: \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.350017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.349987    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.472605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.469945    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88jq\" (UniqueName: \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.472605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.470132    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.472605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.470225    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.472605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.470332    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67fg\" (UniqueName: \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.472605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.471699    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.538558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.522686    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88jq\" (UniqueName: \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\") pod \"payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.596553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.576839    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.596553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.577194    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67fg\" (UniqueName: \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.596553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.578729    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.596553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.591498    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67fg\" (UniqueName: \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\") pod \"payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.604554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.600753    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:23.646559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.640086    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:23.939058 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.939029    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl" event={"ID":"9266799c-03ef-4db5-b2f9-04dd5bdc44f1","Type":"ContainerDied","Data":"ffff9a719db777fce367ded5025e3eb9fe2dbda7d86a2e0d3f0a2fdfee1cf6b1"}
Nov 02 22:19:23.939496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.939480    4929 scope.go:122] "RemoveContainer" containerID="249788cc2fb9a0c2ab35c1f3313be3d0cceebce169c46c408f96812c33559283"
Nov 02 22:19:23.939728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.939711    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"
Nov 02 22:19:23.988301 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.988232    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:19:23.997781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:23.996509    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14dae2-84784df74c-mmptl"]
Nov 02 22:19:24.113714 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:24.113678    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:24.160989 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:24.160955    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3fbabf_7548_40ac_a0d1_b354f0e0e3f7.slice/crio-49e84bb8654bf5703e4adbcd4094eb4ae77569c53e59f017cc418c1089945fec WatchSource:0}: Error finding container 49e84bb8654bf5703e4adbcd4094eb4ae77569c53e59f017cc418c1089945fec: Status 404 returned error can't find the container with id 49e84bb8654bf5703e4adbcd4094eb4ae77569c53e59f017cc418c1089945fec
Nov 02 22:19:24.162256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.162229    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:19:24.221769 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:24.221732    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f5f30_52a2_4af0_9375_128cac11fe8e.slice/crio-a11ceaf33b94f5eaa2487b5ca1c652040985bed37803b282c16d9379ad5d92d9 WatchSource:0}: Error finding container a11ceaf33b94f5eaa2487b5ca1c652040985bed37803b282c16d9379ad5d92d9: Status 404 returned error can't find the container with id a11ceaf33b94f5eaa2487b5ca1c652040985bed37803b282c16d9379ad5d92d9
Nov 02 22:19:24.223139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.223091    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:19:24.808667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.808633    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9266799c-03ef-4db5-b2f9-04dd5bdc44f1" path="/var/lib/kubelet/pods/9266799c-03ef-4db5-b2f9-04dd5bdc44f1/volumes"
Nov 02 22:19:24.985347 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.985270    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" event={"ID":"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7","Type":"ContainerStarted","Data":"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"}
Nov 02 22:19:24.985949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.985922    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" event={"ID":"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7","Type":"ContainerStarted","Data":"49e84bb8654bf5703e4adbcd4094eb4ae77569c53e59f017cc418c1089945fec"}
Nov 02 22:19:24.986696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.986596    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:24.989642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.989606    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" event={"ID":"ed8f5f30-52a2-4af0-9375-128cac11fe8e","Type":"ContainerStarted","Data":"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"}
Nov 02 22:19:24.989775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.989661    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" event={"ID":"ed8f5f30-52a2-4af0-9375-128cac11fe8e","Type":"ContainerStarted","Data":"a11ceaf33b94f5eaa2487b5ca1c652040985bed37803b282c16d9379ad5d92d9"}
Nov 02 22:19:24.989775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:24.989765    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:25.006896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:25.006851    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" podStartSLOduration=2.006838596 podStartE2EDuration="2.006838596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:25.002526989 +0000 UTC m=+1302.310617794" watchObservedRunningTime="2026-11-02 22:19:25.006838596 +0000 UTC m=+1302.314929450"
Nov 02 22:19:25.021110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:25.021063    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" podStartSLOduration=2.021050209 podStartE2EDuration="2.021050209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:25.018668383 +0000 UTC m=+1302.326759190" watchObservedRunningTime="2026-11-02 22:19:25.021050209 +0000 UTC m=+1302.329141013"
Nov 02 22:19:25.152968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:25.152935    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:19:25.153232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:25.153201    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" containerID="cri-o://7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1" gracePeriod=30
Nov 02 22:19:25.345800 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:25.331109    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:25.361566 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:25.361075    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:26.930951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:26.930910    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:19:26.958612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:26.953358    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:19:27.498495 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:27.498459    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:19:28.365978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:28.365938    4929 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:19:28.366363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:28.366013    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:19:30.468507 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.468478    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:19:30.549997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.549963    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550030    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550053    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550089    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550286    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550324    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:30.550478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550370    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550448    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550647    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:30.550765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550736    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:30.550865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550748    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.550969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.550946    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\") pod \"84d60ac4-573a-4fb2-988b-589720c87aee\" (UID: \"84d60ac4-573a-4fb2-988b-589720c87aee\") "
Nov 02 22:19:30.551201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.551178    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:30.551298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.551277    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:30.553014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.552491    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.553014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.552516    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.553014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.552761    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.553014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.552781    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.553014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.552795    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84d60ac4-573a-4fb2-988b-589720c87aee-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.553855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.553830    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:30.553945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.553901    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:30.554496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.554473    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:19:30.555227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.555205    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88" pod "84d60ac4-573a-4fb2-988b-589720c87aee" (UID: "84d60ac4-573a-4fb2-988b-589720c87aee"). InnerVolumeSpecName "kube-api-access-t5w88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:30.654213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.654172    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5w88\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-kube-api-access-t5w88\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.654213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.654213    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84d60ac4-573a-4fb2-988b-589720c87aee-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.654434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.654231    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d60ac4-573a-4fb2-988b-589720c87aee-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:30.654434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:30.654243    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84d60ac4-573a-4fb2-988b-589720c87aee-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:31.132111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.132077    4929 generic.go:356] "Generic (PLEG): container finished" podID="84d60ac4-573a-4fb2-988b-589720c87aee" containerID="7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1" exitCode=0
Nov 02 22:19:31.132425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.132133    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"
Nov 02 22:19:31.132425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.132154    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerDied","Data":"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"}
Nov 02 22:19:31.132425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.132215    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n" event={"ID":"84d60ac4-573a-4fb2-988b-589720c87aee","Type":"ContainerDied","Data":"4d92b4448d465ecc10646f2b6d228f0661a8afb9255d5e46e449731adb993fde"}
Nov 02 22:19:31.132425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.132242    4929 scope.go:122] "RemoveContainer" containerID="7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"
Nov 02 22:19:31.145444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.145417    4929 scope.go:122] "RemoveContainer" containerID="20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"
Nov 02 22:19:31.153225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.153129    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:19:31.158082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.158045    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-z759n"]
Nov 02 22:19:31.167400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.167371    4929 scope.go:122] "RemoveContainer" containerID="7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"
Nov 02 22:19:31.167939 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:31.167909    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1\": container with ID starting with 7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1 not found: ID does not exist" containerID="7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"
Nov 02 22:19:31.168043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.167992    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1"} err="failed to get container status \"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1\": rpc error: code = NotFound desc = could not find container \"7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1\": container with ID starting with 7c535af8f43161e17863229915859a39ce0cecc1494256e2fb05fed9f79753e1 not found: ID does not exist"
Nov 02 22:19:31.168043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.168023    4929 scope.go:122] "RemoveContainer" containerID="20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"
Nov 02 22:19:31.168866 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:31.168845    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc\": container with ID starting with 20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc not found: ID does not exist" containerID="20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"
Nov 02 22:19:31.168928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:31.168873    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc"} err="failed to get container status \"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc\": rpc error: code = NotFound desc = could not find container \"20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc\": container with ID starting with 20bfb820ae34b558449fcc172635453f9ec733b0e6e5948ef3c450efab8029fc not found: ID does not exist"
Nov 02 22:19:32.597454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:32.597417    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:19:32.815654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:32.815621    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" path="/var/lib/kubelet/pods/84d60ac4-573a-4fb2-988b-589720c87aee/volumes"
Nov 02 22:19:34.216717 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:34.216672    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:34.671628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.671588    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:19:34.672840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.672796    4929 cpu_manager.go:414] "containerMap: removing container" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.672840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.672822    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.672840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.672838    4929 cpu_manager.go:414] "containerMap: removing container" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.672840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.672847    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.673240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.673030    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.673240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.673046    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d60ac4-573a-4fb2-988b-589720c87aee" containerName="istio-proxy"
Nov 02 22:19:34.677313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.677286    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.682647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.682620    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-419f68\""
Nov 02 22:19:34.682810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.682789    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-419f68-dockercfg-7hkz4\""
Nov 02 22:19:34.682903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.682827    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:19:34.708575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.708528    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:19:34.720425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.717331    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:34.736584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.728932    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:19:34.815559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.813226    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:34.815559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.813291    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn5b\" (UniqueName: \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.815559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.813834    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvx8h\" (UniqueName: \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:34.815559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.813986    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.956202    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.956437    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.956481    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn5b\" (UniqueName: \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.956764    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvx8h\" (UniqueName: \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.958053    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:34.962611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:34.958925    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:35.246724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:35.227714    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn5b\" (UniqueName: \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\") pod \"payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:35.246724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:35.228376    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvx8h\" (UniqueName: \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\") pod \"payload-processing-e2e-inf-a-419f68-f944c6b85-76t78\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:35.258853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:35.251308    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:35.305104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:35.303009    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:35.913217 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:35.913178    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04565c1_4caa_4c23_b750_4cf05ffdcf90.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:36.000742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.000701    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:19:36.112631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.112601    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:19:36.117591 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:36.117521    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a017c56_f362_43c3_9efd_3461e8dcb554.slice/crio-4dcded819bc6de1a062f5c2ff81afadf34f0d0c5fea185f38b4a699b9cbddec9 WatchSource:0}: Error finding container 4dcded819bc6de1a062f5c2ff81afadf34f0d0c5fea185f38b4a699b9cbddec9: Status 404 returned error can't find the container with id 4dcded819bc6de1a062f5c2ff81afadf34f0d0c5fea185f38b4a699b9cbddec9
Nov 02 22:19:36.118597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.118576    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:19:36.531387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.531295    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" event={"ID":"6a017c56-f362-43c3-9efd-3461e8dcb554","Type":"ContainerStarted","Data":"1a38fd7a14779c1c6ec53ba9edfdfbe6ccc770def2ffa7d1f36bab54ce733978"}
Nov 02 22:19:36.532763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.532735    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" event={"ID":"6a017c56-f362-43c3-9efd-3461e8dcb554","Type":"ContainerStarted","Data":"4dcded819bc6de1a062f5c2ff81afadf34f0d0c5fea185f38b4a699b9cbddec9"}
Nov 02 22:19:36.532997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.532976    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:36.534528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.534506    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" event={"ID":"123aac54-2085-49f4-8e27-483ccf274a1e","Type":"ContainerStarted","Data":"47573edb14c6dbb56412cc050cc9348b6ec0f42b4ba77d7a83c01bb07329ad0b"}
Nov 02 22:19:36.534676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.534634    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" event={"ID":"123aac54-2085-49f4-8e27-483ccf274a1e","Type":"ContainerStarted","Data":"43b1d673e1d8e374805f18331f7f41e792937e968ed5ab592e9bcbf4922365cc"}
Nov 02 22:19:36.534807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.534787    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:36.549690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.549643    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" podStartSLOduration=2.549629667 podStartE2EDuration="2.549629667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:36.546806391 +0000 UTC m=+1313.854897196" watchObservedRunningTime="2026-11-02 22:19:36.549629667 +0000 UTC m=+1313.857720469"
Nov 02 22:19:36.565394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:36.565345    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" podStartSLOduration=2.565332058 podStartE2EDuration="2.565332058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:36.564790382 +0000 UTC m=+1313.872881187" watchObservedRunningTime="2026-11-02 22:19:36.565332058 +0000 UTC m=+1313.873422863"
Nov 02 22:19:37.062318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:37.062285    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:19:37.597904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:37.597865    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing" containerID="cri-o://b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1" gracePeriod=25
Nov 02 22:19:41.848987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:41.848954    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:19:42.018819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.018756    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" containerID="cri-o://bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c" gracePeriod=45
Nov 02 22:19:42.033456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.033399    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" containerID="cri-o://8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba" gracePeriod=45
Nov 02 22:19:42.100242 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.100176    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Nov 02 22:19:42.370817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.370788    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:19:42.375789 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.375760    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:19:42.556206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556174    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.556384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556245    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.556384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556270    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.556505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556406    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.556505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556428    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.556663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556519    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.556663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556624    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.556764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556669    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.556764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556698    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.556764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556725    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\" (UniqueName: \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.556920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556838    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\" (UniqueName: \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.556920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556860    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.556920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556894    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\") pod \"23fa1f86-dd73-4b71-8eeb-4229428a8649\" (UID: \"23fa1f86-dd73-4b71-8eeb-4229428a8649\") "
Nov 02 22:19:42.557082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556921    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.557082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556928    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.557082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.556999    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\") pod \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\" (UID: \"bfe65aff-64bc-4387-a4ae-46f4064ad0ce\") "
Nov 02 22:19:42.557254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557166    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.557254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557212    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.557922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557891    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.557922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557924    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.558124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557939    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.558124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557952    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.558124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557965    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.558124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.557978    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.559119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.559092    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:19:42.559265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.559223    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:19:42.559473 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.559445    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "kube-api-access-wvb2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:42.559473 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.559467    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm" pod "bfe65aff-64bc-4387-a4ae-46f4064ad0ce" (UID: "bfe65aff-64bc-4387-a4ae-46f4064ad0ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.559642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.559524    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:42.560121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.560093    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb" pod "23fa1f86-dd73-4b71-8eeb-4229428a8649" (UID: "23fa1f86-dd73-4b71-8eeb-4229428a8649"). InnerVolumeSpecName "kube-api-access-mtgdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:42.652608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.651622    4929 generic.go:356] "Generic (PLEG): container finished" podID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerID="bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c" exitCode=0
Nov 02 22:19:42.652608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.651972    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" event={"ID":"23fa1f86-dd73-4b71-8eeb-4229428a8649","Type":"ContainerDied","Data":"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"}
Nov 02 22:19:42.652608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.652020    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" event={"ID":"23fa1f86-dd73-4b71-8eeb-4229428a8649","Type":"ContainerDied","Data":"d105e65892a48ba8f8b31509eceedfb028e990259462f072ac7e02dea63025d4"}
Nov 02 22:19:42.652608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.652043    4929 scope.go:122] "RemoveContainer" containerID="bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"
Nov 02 22:19:42.652608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.652229    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"
Nov 02 22:19:42.679621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.676511    4929 generic.go:356] "Generic (PLEG): container finished" podID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerID="8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba" exitCode=0
Nov 02 22:19:42.679621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.676694    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" event={"ID":"bfe65aff-64bc-4387-a4ae-46f4064ad0ce","Type":"ContainerDied","Data":"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"}
Nov 02 22:19:42.679621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.676741    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct" event={"ID":"bfe65aff-64bc-4387-a4ae-46f4064ad0ce","Type":"ContainerDied","Data":"67449cd295a1fd22622e7458b74dd1b1b8d08bfe3cab17955ac6355eb18469a3"}
Nov 02 22:19:42.679621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.676821    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687097    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa1f86-dd73-4b71-8eeb-4229428a8649-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687132    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvb2v\" (UniqueName: \"kubernetes.io/projected/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-kube-api-access-wvb2v\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687150    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgdb\" (UniqueName: \"kubernetes.io/projected/23fa1f86-dd73-4b71-8eeb-4229428a8649-kube-api-access-mtgdb\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687164    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687179    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe65aff-64bc-4387-a4ae-46f4064ad0ce-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.696745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.687191    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa1f86-dd73-4b71-8eeb-4229428a8649-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:42.738398 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.734248    4929 scope.go:122] "RemoveContainer" containerID="bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"
Nov 02 22:19:42.741154 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:42.740741    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c\": container with ID starting with bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c not found: ID does not exist" containerID="bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"
Nov 02 22:19:42.741154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.740784    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c"} err="failed to get container status \"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c\": rpc error: code = NotFound desc = could not find container \"bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c\": container with ID starting with bbf35e3f0ed3284b0fe01f8bfded69e98f56b970e04bab735f1531e0efd7169c not found: ID does not exist"
Nov 02 22:19:42.741154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.740809    4929 scope.go:122] "RemoveContainer" containerID="8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"
Nov 02 22:19:42.765254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.765213    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:19:42.770784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.770646    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb"]
Nov 02 22:19:42.774301 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.774279    4929 scope.go:122] "RemoveContainer" containerID="8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"
Nov 02 22:19:42.774801 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:42.774768    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba\": container with ID starting with 8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba not found: ID does not exist" containerID="8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"
Nov 02 22:19:42.775651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.775620    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba"} err="failed to get container status \"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba\": rpc error: code = NotFound desc = could not find container \"8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba\": container with ID starting with 8a266b4579f3483bb43a1aef2f6d7f8f524a4e89c72e5d6124276ae5d8517dba not found: ID does not exist"
Nov 02 22:19:42.790019 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.786665    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:19:42.797056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.797028    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/premium-w1-5f4a2f-kserve-6cdcf4c68f-kg8ct"]
Nov 02 22:19:42.810564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.810517    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" path="/var/lib/kubelet/pods/23fa1f86-dd73-4b71-8eeb-4229428a8649/volumes"
Nov 02 22:19:42.811574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:42.811528    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" path="/var/lib/kubelet/pods/bfe65aff-64bc-4387-a4ae-46f4064ad0ce/volumes"
Nov 02 22:19:43.325333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:43.321638    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-w1-5f4a2f-kserve-765f7c686d-gk2pb" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:45.425112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:45.425077    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:19:45.425557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:45.425282    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" containerID="cri-o://7b6393c6eb2fcfc566a5755b1d7d945305c32be83170bac0e9167204491435f4" gracePeriod=30
Nov 02 22:19:47.227632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.227595    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:19:47.239988 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.239945    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:19:47.591613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.583175    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:19:47.591613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.583226    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:19:47.591613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.584610    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:19:47.639559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:47.638896    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:19:48.589558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:48.587044    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:19:51.759582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:51.732002    4929 generic.go:356] "Generic (PLEG): container finished" podID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerID="7b6393c6eb2fcfc566a5755b1d7d945305c32be83170bac0e9167204491435f4" exitCode=0
Nov 02 22:19:51.759582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:51.737383    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" event={"ID":"0ea2f692-29d4-4cff-b852-1e3199d41c6f","Type":"ContainerDied","Data":"7b6393c6eb2fcfc566a5755b1d7d945305c32be83170bac0e9167204491435f4"}
Nov 02 22:19:52.063158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.063054    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:19:52.157257 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.157209    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.162797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.162669    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:52.163179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.163151    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179307    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179406    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179503    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179660    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179783    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179886    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.180603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.179927    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\") pod \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\" (UID: \"0ea2f692-29d4-4cff-b852-1e3199d41c6f\") "
Nov 02 22:19:52.187888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.183245    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:52.188403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.188188    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.189238 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.188970    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:52.189561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.189410    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:52.189876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.189826    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:52.244337 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.243064    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing" containerID="cri-o://4e55ee743b68e1567afbc5c4b413e9ecb3f0736128fdd58fc030eacf0eddbe5e" gracePeriod=25
Nov 02 22:19:52.355206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.313971    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.429762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.429730    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.430006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.429990    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.430106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.430093    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.430187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.370736    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:52.627613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.627499    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "kube-api-access-xlg2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:52.634253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.631816    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.634253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.631849    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlg2q\" (UniqueName: \"kubernetes.io/projected/0ea2f692-29d4-4cff-b852-1e3199d41c6f-kube-api-access-xlg2q\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.661558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.655755    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:19:52.661558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.655843    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo" pod "0ea2f692-29d4-4cff-b852-1e3199d41c6f" (UID: "0ea2f692-29d4-4cff-b852-1e3199d41c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:19:52.667295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.667252    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:52.668762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.668732    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:19:52.672556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.669136    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" containerID="cri-o://445e839b1cf592918352614c41549bfc2ef9a945392d8e333907103434a52c13" gracePeriod=30
Nov 02 22:19:52.763562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.751443    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.764123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.764087    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2f692-29d4-4cff-b852-1e3199d41c6f-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:52.850914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:52.850855    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:53.002518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.002279    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing" containerID="cri-o://ff85df32a437a53e6c13768b95052a204e45bb140105090532e9d6b19298da0d" gracePeriod=25
Nov 02 22:19:53.025166 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.025134    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"
Nov 02 22:19:53.144559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.117279    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5" event={"ID":"0ea2f692-29d4-4cff-b852-1e3199d41c6f","Type":"ContainerDied","Data":"1bb2a1d03818b90c4bdc4268aded7b631c036642174e4a0b50d80faa679171e8"}
Nov 02 22:19:53.209557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.186591    4929 scope.go:122] "RemoveContainer" containerID="7b6393c6eb2fcfc566a5755b1d7d945305c32be83170bac0e9167204491435f4"
Nov 02 22:19:53.661943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.661888    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:53.684629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.671836    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:19:53.698606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.685253    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:53.773590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:53.773505    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:19:54.062060 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.062004    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:54.114796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.114745    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.196867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.196822    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.198214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.198182    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:54.231689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.231591    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:54.374607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.263549    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.393162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.322653    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.135:50052\" within 1s: context deadline exceeded"
Nov 02 22:19:54.433576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.396745    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.437607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.415379    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.511300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.449739    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:19:54.511300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.498429    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:19:54.616558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.584750    4929 cpu_manager.go:414] "containerMap: removing container" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main"
Nov 02 22:19:54.616833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.616803    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main"
Nov 02 22:19:54.617098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.617078    4929 cpu_manager.go:414] "containerMap: removing container" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy"
Nov 02 22:19:54.617213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.617198    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy"
Nov 02 22:19:54.617319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.617306    4929 cpu_manager.go:414] "containerMap: removing container" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main"
Nov 02 22:19:54.617412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.617401    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main"
Nov 02 22:19:54.667250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.667215    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" containerName="istio-proxy"
Nov 02 22:19:54.667499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.667481    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe65aff-64bc-4387-a4ae-46f4064ad0ce" containerName="main"
Nov 02 22:19:54.667663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.667651    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fa1f86-dd73-4b71-8eeb-4229428a8649" containerName="main"
Nov 02 22:19:54.717557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.685697    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.717557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.698651    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 22:19:54.835514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.835417    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:54.864422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.864381    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:19:55.117721 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.936810    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:19:55.127030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.126995    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:19:55.128128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:54.953515    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.133686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.133206    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.142619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.140970    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.216908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.216861    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.246807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.220082    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.445683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.424622    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.452267    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.452498    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.452652    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.452738    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.452789    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7vd\" (UniqueName: \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.453018    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.453408    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e30968\"/\"openshift-service-ca.crt\""
Nov 02 22:19:55.476560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.453759    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e30968\"/\"default-dockercfg-67vjg\""
Nov 02 22:19:55.478458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.478427    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e30968\"/\"kube-root-ca.crt\""
Nov 02 22:19:55.478882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.478864    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e30968\"/\"test-model-e30968-kserve-self-signed-certs\""
Nov 02 22:19:55.552843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.552789    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:19:55.795629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.795480    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:55.964558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.934363    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:56.029576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.993882    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:56.029576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:55.994323    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:19:56.048106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.047993    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.115730    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121265    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121340    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121524    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121684    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121749    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.121798    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7vd\" (UniqueName: \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.122239    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.122548    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.136559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.122797    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.180766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.147783    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:56.180766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.166226    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.186607 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:56.186571    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea2f692_29d4_4cff_b852_1e3199d41c6f.slice/crio-1bb2a1d03818b90c4bdc4268aded7b631c036642174e4a0b50d80faa679171e8\": RecentStats: unable to find data in memory cache]"
Nov 02 22:19:56.202256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.202215    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:56.221837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.221793    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:19:56.296766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.266732    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:19:56.317262 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:56.298110    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.171s"
Nov 02 22:19:56.321598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.317522    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:19:56.474435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.412127    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.442156    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497377    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497478    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497499    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497696    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497735    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497771    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497809    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:19:56.500574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.497840    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:19:56.643560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.613764    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.857603    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.858408    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:56.886196    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7vd\" (UniqueName: \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\") pod \"test-model-e30968-kserve-5bf44668b9-8gjnz\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") " pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.113635    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" containerID="cri-o://0bf45d1e3fcac438ef014afa02a8b0aa631f45289d93db7e723bef4d95a03edb" gracePeriod=45
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.155161    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:19:57.204114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.127613    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.223568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.172681    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:19:57.233641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.172729    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.406765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.406715    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.431393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.431357    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:19:57.476722 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.476602    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:57.713690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.713641    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:19:58.181636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.181578    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" podUID="7a4a8e4d-2131-4854-9d0b-5010dae227bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:58.247693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.247642    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:58.263566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.261366    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:58.263566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:57.783486    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": read tcp 10.133.0.2:55120->10.133.0.66:8000: read: connection reset by peer"
Nov 02 22:19:58.339372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.323372    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" podUID="bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:58.486685 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:19:57.927100    4929 logging.go:55] [core] [Channel #2401 SubChannel #2403]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:50052", ServerName: "10.133.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:50052: connect: connection refused"
Nov 02 22:19:58.540557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.534297    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Nov 02 22:19:58.540557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.534639    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:19:58.540557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.534968    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:19:58.712229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.701709    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 02 22:19:58.712229 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:19:58.703007    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.845s"
Nov 02 22:19:58.712229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.703040    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f78m5"]
Nov 02 22:19:58.712229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.703179    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:19:58.750772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.750628    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:19:58.776568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.751208    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:19:58.796368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.796320    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 02 22:19:58.832918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:58.832869    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:19:59.125573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.123621    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 02 22:19:59.318444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.318400    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.46:50052\" within 1s: context deadline exceeded"
Nov 02 22:20:00.329874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.329827    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.429220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.429114    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" podUID="7a4a8e4d-2131-4854-9d0b-5010dae227bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.429479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.469990    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.430693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.430659    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.459564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.568750    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.459564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.570311    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.459564 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:00.193178    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:20:00.507073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.451880    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:20:00.605504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.605455    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj" podUID="bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.671615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.662065    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 02 22:20:00.735824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.735715    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 22:20:00.767655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.767609    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.768283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.768251    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:20:00.801113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:19:59.319177    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 22:20:00.833188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.506843    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.913630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.912136    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.979565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.922311    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.136:50052: i/o timeout"
Nov 02 22:20:00.979565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.932269    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc" podUID="8b76db34-a58f-4b26-91c1-d5c678aec811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:00.979565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.940720    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 02 22:20:00.979565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:00.956184    4929 generic.go:356] "Generic (PLEG): container finished" podID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerID="b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1" exitCode=0
Nov 02 22:20:01.127479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.127435    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea2f692-29d4-4cff-b852-1e3199d41c6f" path="/var/lib/kubelet/pods/0ea2f692-29d4-4cff-b852-1e3199d41c6f/volumes"
Nov 02 22:20:01.128520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.128477    4929 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:01.133514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.133486    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:20:01.361653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.343295    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\") pod \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") "
Nov 02 22:20:01.365558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.362125    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\" (UniqueName: \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\") pod \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\" (UID: \"35533e2b-61e1-43eb-9c57-cea4dfeea71f\") "
Nov 02 22:20:01.533655 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.464870    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:20:01.713505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.681675    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 02 22:20:01.765702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.548606    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:01.793756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.793700    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 22:20:01.835561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.817360    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 22:20:01.836172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:01.836107    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:02.086597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.086489    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume" pod "35533e2b-61e1-43eb-9c57-cea4dfeea71f" (UID: "35533e2b-61e1-43eb-9c57-cea4dfeea71f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:02.117602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.103762    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk" pod "35533e2b-61e1-43eb-9c57-cea4dfeea71f" (UID: "35533e2b-61e1-43eb-9c57-cea4dfeea71f"). InnerVolumeSpecName "kube-api-access-5rwxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:02.117785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.112216    4929 generic.go:356] "Generic (PLEG): container finished" podID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerID="0bf45d1e3fcac438ef014afa02a8b0aa631f45289d93db7e723bef4d95a03edb" exitCode=0
Nov 02 22:20:02.126236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.120097    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm" podUID="7a4a8e4d-2131-4854-9d0b-5010dae227bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:02.126236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.123014    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35533e2b-61e1-43eb-9c57-cea4dfeea71f-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.126236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.123037    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rwxk\" (UniqueName: \"kubernetes.io/projected/35533e2b-61e1-43eb-9c57-cea4dfeea71f-kube-api-access-5rwxk\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.127218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.127184    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:02.127950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.127918    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:02.205199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.205154    4929 generic.go:356] "Generic (PLEG): container finished" podID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerID="445e839b1cf592918352614c41549bfc2ef9a945392d8e333907103434a52c13" exitCode=0
Nov 02 22:20:02.210000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.209961    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:02.212221 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:02.212196    4929 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.084s"
Nov 02 22:20:02.212396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212381    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:20:02.212456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212427    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:20:02.212514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212458    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:20:02.212514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212499    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:20:02.212634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212550    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:20:02.212634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212575    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" event={"ID":"35533e2b-61e1-43eb-9c57-cea4dfeea71f","Type":"ContainerDied","Data":"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"}
Nov 02 22:20:02.212634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212622    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:20:02.212791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212662    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:20:02.212791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212678    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" event={"ID":"b08deb0f-241c-478a-b608-d9409bc3c8d8","Type":"ContainerDied","Data":"0bf45d1e3fcac438ef014afa02a8b0aa631f45289d93db7e723bef4d95a03edb"}
Nov 02 22:20:02.212791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212693    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerDied","Data":"445e839b1cf592918352614c41549bfc2ef9a945392d8e333907103434a52c13"}
Nov 02 22:20:02.212791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212757    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:20:02.212986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212808    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:20:02.212986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.212842    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-svvlm"
Nov 02 22:20:02.215217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.215193    4929 scope.go:122] "RemoveContainer" containerID="b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"
Nov 02 22:20:02.227343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.227314    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:20:02.227476 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.227416    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:20:02.233056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.233026    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" event={"ID":"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd","Type":"ContainerStarted","Data":"a9d9a4fafe79faaf66e03a36389771043f82826f4d236dd2da4f29b414ea714e"}
Nov 02 22:20:02.292786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.286010    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b" event={"ID":"35533e2b-61e1-43eb-9c57-cea4dfeea71f","Type":"ContainerDied","Data":"29e5af6f64231016c3d06761818aa4eb450d6fc893a609410d91ae3e6f7044a8"}
Nov 02 22:20:02.292786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.286086    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"
Nov 02 22:20:02.293500 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.293462    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:20:02.299855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.299834    4929 scope.go:122] "RemoveContainer" containerID="d42dac41d7c4a660ebc0db959be99516219fc984ea974b74f4b96cd006b89221"
Nov 02 22:20:02.311688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.306907    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:20:02.375093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.357703    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:20:02.376664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.375884    4929 scope.go:122] "RemoveContainer" containerID="b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"
Nov 02 22:20:02.376664 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:02.376375    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1\": container with ID starting with b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1 not found: ID does not exist" containerID="b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"
Nov 02 22:20:02.376664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.376413    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1"} err="failed to get container status \"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1\": rpc error: code = NotFound desc = could not find container \"b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1\": container with ID starting with b312405d1e27bb88d113c410c1a425a694f2eb03ae2cfc0b66b6909ed38e4da1 not found: ID does not exist"
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443039    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443106    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443185    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443224    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443301    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\" (UniqueName: \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443350    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443401    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443454    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443601    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443650    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443746    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443788    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443834    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443871    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\") pod \"30f87b95-4f7f-4c53-863a-6a73687c12eb\" (UID: \"30f87b95-4f7f-4c53-863a-6a73687c12eb\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.443978    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\") pod \"b08deb0f-241c-478a-b608-d9409bc3c8d8\" (UID: \"b08deb0f-241c-478a-b608-d9409bc3c8d8\") "
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.444886    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.445731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.445339    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.473960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.473660    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.477179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.477143    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.479311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.479281    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.480412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.480101    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.480957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.480934    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.481285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.481265    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.481807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.481778    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:02.482147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.482121    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.482616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.482521    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:02.484231 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.484206    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "kube-api-access-hc4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:02.484652 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.484617    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs" pod "b08deb0f-241c-478a-b608-d9409bc3c8d8" (UID: "b08deb0f-241c-478a-b608-d9409bc3c8d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:20:02.484925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.484894    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:02.486317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.485566    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "kube-api-access-xxxhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:02.486317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.485683    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo" pod "30f87b95-4f7f-4c53-863a-6a73687c12eb" (UID: "30f87b95-4f7f-4c53-863a-6a73687c12eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:02.520246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.520220    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:20:02.526235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.526197    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f4a2f-56cc574c49-82j9b"]
Nov 02 22:20:02.551400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.551363    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.551654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.551637    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08deb0f-241c-478a-b608-d9409bc3c8d8-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.551765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.551752    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.551859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.551846    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.551972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.551947    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4qd\" (UniqueName: \"kubernetes.io/projected/b08deb0f-241c-478a-b608-d9409bc3c8d8-kube-api-access-hc4qd\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552063    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552168 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552154    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxhr\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-kube-api-access-xxxhr\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552249    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552337    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f87b95-4f7f-4c53-863a-6a73687c12eb-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552423    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552514    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552637    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552783    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f87b95-4f7f-4c53-863a-6a73687c12eb-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.552916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.552903    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08deb0f-241c-478a-b608-d9409bc3c8d8-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:02.969984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:02.958713    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" path="/var/lib/kubelet/pods/35533e2b-61e1-43eb-9c57-cea4dfeea71f/volumes"
Nov 02 22:20:03.178930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.178893    4929 patch_prober.go:29] interesting pod/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:20:03.179139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.178957    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:03.298847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.298765    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"
Nov 02 22:20:03.298847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.298801    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg" event={"ID":"b08deb0f-241c-478a-b608-d9409bc3c8d8","Type":"ContainerDied","Data":"fdc77fe0d76e8dc9b9e0e20889d3f87c14194fabcb54af1ee7658e615aa0f57a"}
Nov 02 22:20:03.299087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.298860    4929 scope.go:122] "RemoveContainer" containerID="0bf45d1e3fcac438ef014afa02a8b0aa631f45289d93db7e723bef4d95a03edb"
Nov 02 22:20:03.300747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.300725    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"
Nov 02 22:20:03.300747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.300731    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w" event={"ID":"30f87b95-4f7f-4c53-863a-6a73687c12eb","Type":"ContainerDied","Data":"a49d483bc44e95f6b02c3e7e214a75f8322d36403c292887619ed9ae6de08e40"}
Nov 02 22:20:03.303157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.303136    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" event={"ID":"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd","Type":"ContainerStarted","Data":"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"}
Nov 02 22:20:03.311153 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.311135    4929 scope.go:122] "RemoveContainer" containerID="445e839b1cf592918352614c41549bfc2ef9a945392d8e333907103434a52c13"
Nov 02 22:20:03.345110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.345049    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" podStartSLOduration=9.34503455 podStartE2EDuration="9.34503455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:03.34402982 +0000 UTC m=+1340.652120626" watchObservedRunningTime="2026-11-02 22:20:03.34503455 +0000 UTC m=+1340.653125359"
Nov 02 22:20:03.363513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.363475    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:20:03.371875 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.371831    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14dae2-openshift-default-55696bc79c-jxw4w"]
Nov 02 22:20:03.386754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.386724    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:20:03.393843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.393812    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-2-w6-a1d78c-kserve-7dd5486f54-ng5zg"]
Nov 02 22:20:03.432302 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:03.432272    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:04.316479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.316437    4929 generic.go:356] "Generic (PLEG): container finished" podID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerID="2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79" exitCode=2
Nov 02 22:20:04.316695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.316626    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerDied","Data":"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"}
Nov 02 22:20:04.317248 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.317224    4929 scope.go:122] "RemoveContainer" containerID="2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"
Nov 02 22:20:04.327076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.319621    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:04.327076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.319705    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:04.844042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.840475    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" path="/var/lib/kubelet/pods/30f87b95-4f7f-4c53-863a-6a73687c12eb/volumes"
Nov 02 22:20:04.846018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:04.845911    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" path="/var/lib/kubelet/pods/b08deb0f-241c-478a-b608-d9409bc3c8d8/volumes"
Nov 02 22:20:05.340206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.340156    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerStarted","Data":"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"}
Nov 02 22:20:05.341849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.341687    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:20:05.342476 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.342449    4929 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k8bzv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:05.342632 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.342498    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:05.563125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.562702    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:20:05.563417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.563385    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:20:05.795640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.756641    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:20:05.795640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:05.756716    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:06.349079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.349025    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:20:06.694619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.694575    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:20:06.694861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.694833    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" containerID="cri-o://450f7e31105a3c9c69762f3f9520933f24c027ecf0519e987fc71ac8c11e4803" gracePeriod=30
Nov 02 22:20:06.698039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.698019    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:20:06.698455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.698425    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" containerID="cri-o://88b5f3ccb9203c86849451b823d7452f535268153df846770025bfa5ac811031" gracePeriod=30
Nov 02 22:20:06.709386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.709356    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:20:06.709638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.709601    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" containerID="cri-o://2afa3e4b077949cf03bc8b98e06ec0c992af7de15847f59e0d50d37ff56ba67d" gracePeriod=30
Nov 02 22:20:06.718853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.718827    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:20:06.720991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.720957    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" containerID="cri-o://607b6ae88b9c6178191dd0f5001967b857d8795cf2864cecdb0334da4a6d444d" gracePeriod=30
Nov 02 22:20:06.966931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.965424    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:06.966931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:06.965508    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:07.432096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:07.432051    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:07.445545 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:07.445497    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:07.692867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:07.692789    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:07.692867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:07.692848    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.132766    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134591    4929 cpu_manager.go:414] "containerMap: removing container" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134615    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134655    4929 cpu_manager.go:414] "containerMap: removing container" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134668    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134706    4929 cpu_manager.go:414] "containerMap: removing container" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.134714    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.135142    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="35533e2b-61e1-43eb-9c57-cea4dfeea71f" containerName="payload-processing"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.135165    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.136580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.135203    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08deb0f-241c-478a-b608-d9409bc3c8d8" containerName="main"
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.141598    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.142736    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.145136    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-00c72a\"/\"openshift-service-ca.crt\""
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.146445    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-00c72a\"/\"kube-root-ca.crt\""
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.146675    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-00c72a\"/\"default-dockercfg-bnw5q\""
Nov 02 22:20:08.149222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.146692    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-00c72a\"/\"test-model-00c72a-kserve-self-signed-certs\""
Nov 02 22:20:08.176820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.176528    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.176820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.176694    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.176820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.176765    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.183563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.176892    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.183563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.176996    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.183563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.177196    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4wp\" (UniqueName: \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298093    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298208    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298279    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298564    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4wp\" (UniqueName: \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298747    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.298837    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.299466    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.330556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.300217    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.342667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.340386    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.342667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.341323    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.395110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.373591    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:20:08.395110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.374879    4929 cpu_manager.go:414] "containerMap: removing container" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.395110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.374902    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.395110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.375129    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f87b95-4f7f-4c53-863a-6a73687c12eb" containerName="istio-proxy"
Nov 02 22:20:08.427162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.424994    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.438737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.437175    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:20:08.438737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.437306    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.449907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.449410    4929 generic.go:356] "Generic (PLEG): container finished" podID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerID="1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf" exitCode=0
Nov 02 22:20:08.450888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.450670    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerDied","Data":"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"}
Nov 02 22:20:08.463479 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.462402    4929 scope.go:122] "RemoveContainer" containerID="1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"
Nov 02 22:20:08.469643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.467811    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:08.476553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.470854    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:08.476553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.471095    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:08.476553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.471150    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:08.486345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.486307    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4wp\" (UniqueName: \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\") pod \"test-model-00c72a-kserve-5f76bbbd45-cmfnx\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") " pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.505183 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505140    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505216    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505269    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwsxb\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505298    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505359    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505455    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.505514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505500    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.506018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.505885    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.506081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.506021    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.607805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.607775    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.607953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.607844    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwsxb\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.607953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.607892    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.607993    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608053    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608082    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608240    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608305    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608406    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608660    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608701    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.608798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.608757    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.610153 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.610130    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.610880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.610840    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.611246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.611223    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.613954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.613932    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.617179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.617151    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.617800 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.617772    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwsxb\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:08.763959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.763860    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:08.862087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:08.862053    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:09.079806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.079768    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:20:09.080632 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:09.080606    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69074b1c_17c2_4630_9889_7fb1c7b0a8d8.slice/crio-093cb4e0b55c883bd81ec2a00c13992560c2cbab873d99d6370f77c7fd5aa30c WatchSource:0}: Error finding container 093cb4e0b55c883bd81ec2a00c13992560c2cbab873d99d6370f77c7fd5aa30c: Status 404 returned error can't find the container with id 093cb4e0b55c883bd81ec2a00c13992560c2cbab873d99d6370f77c7fd5aa30c
Nov 02 22:20:09.187158 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:09.187129    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8970e7_a632_48d5_9f65_d7adf860c52c.slice/crio-df08e7d160cfff0568a960dd9d9d32d2eabd76af21d8f83c4bfca7d8086fa98a WatchSource:0}: Error finding container df08e7d160cfff0568a960dd9d9d32d2eabd76af21d8f83c4bfca7d8086fa98a: Status 404 returned error can't find the container with id df08e7d160cfff0568a960dd9d9d32d2eabd76af21d8f83c4bfca7d8086fa98a
Nov 02 22:20:09.187275 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.187210    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:20:09.190389 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.190348    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.190494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.190437    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.190494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.190481    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.469180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.469141    4929 generic.go:356] "Generic (PLEG): container finished" podID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerID="14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840" exitCode=0
Nov 02 22:20:09.469897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.469864    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerDied","Data":"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"}
Nov 02 22:20:09.470729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.470705    4929 scope.go:122] "RemoveContainer" containerID="14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"
Nov 02 22:20:09.486519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.486463    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.501835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.496981    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.501835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.497082    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:09.507750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.505297    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" event={"ID":"9f8970e7-a632-48d5-9f65-d7adf860c52c","Type":"ContainerStarted","Data":"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"}
Nov 02 22:20:09.507750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.505367    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" event={"ID":"9f8970e7-a632-48d5-9f65-d7adf860c52c","Type":"ContainerStarted","Data":"df08e7d160cfff0568a960dd9d9d32d2eabd76af21d8f83c4bfca7d8086fa98a"}
Nov 02 22:20:09.536990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.536934    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" event={"ID":"69074b1c-17c2-4630-9889-7fb1c7b0a8d8","Type":"ContainerStarted","Data":"6241bd78d3fb37a018a54adb638847161a88c271a6da3f627cae96b0c7957673"}
Nov 02 22:20:09.536990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.536991    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" event={"ID":"69074b1c-17c2-4630-9889-7fb1c7b0a8d8","Type":"ContainerStarted","Data":"093cb4e0b55c883bd81ec2a00c13992560c2cbab873d99d6370f77c7fd5aa30c"}
Nov 02 22:20:09.550268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.550205    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerStarted","Data":"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"}
Nov 02 22:20:09.585566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.585453    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podStartSLOduration=1.585433036 podStartE2EDuration="1.585433036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:09.5459219 +0000 UTC m=+1346.854012706" watchObservedRunningTime="2026-11-02 22:20:09.585433036 +0000 UTC m=+1346.893523843"
Nov 02 22:20:09.587711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.586292    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" podStartSLOduration=1.586275583 podStartE2EDuration="1.586275583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:09.584771153 +0000 UTC m=+1346.892861985" watchObservedRunningTime="2026-11-02 22:20:09.586275583 +0000 UTC m=+1346.894366389"
Nov 02 22:20:09.761614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.760937    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:20:09.761614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.761113    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:09.761614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.761160    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:09.793570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.786879    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:09.870558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:09.865696    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:10.194607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.194572    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:10.194787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.194624    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:10.262451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.262413    4929 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:10.262679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.262478    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:10.589178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.589086    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerStarted","Data":"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"}
Nov 02 22:20:10.759035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.758997    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:10.759358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.759313    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:10.882561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.881688    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:10.882561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:10.881754    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:11.017560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.008007    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:20:11.682555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.681644    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:20:11.682555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.681810    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:11.682555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.681856    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:11.832595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.792795    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:11.832595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.792860    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:11.887558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.887079    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:11.887558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:11.887137    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:12.254618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.233650    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 22:20:12.291312 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.290735    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:12.338417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.337614    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:12.443575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.430845    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:12.618244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.617740    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:12.618244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.617806    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:12.771586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.768212    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:12.771586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.768268    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:13.015650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.889482    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:13.015650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:12.889565    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:13.135305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.135084    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:13.180339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.179677    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:13.391858 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.375112    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:13.391858 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.384770    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:20:13.523276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.523210    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:13.669068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.651821    4929 generic.go:356] "Generic (PLEG): container finished" podID="165850e3-d69d-414d-906a-67cb59e15405" containerID="2afa3e4b077949cf03bc8b98e06ec0c992af7de15847f59e0d50d37ff56ba67d" exitCode=0
Nov 02 22:20:13.669068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.652163    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" event={"ID":"165850e3-d69d-414d-906a-67cb59e15405","Type":"ContainerDied","Data":"2afa3e4b077949cf03bc8b98e06ec0c992af7de15847f59e0d50d37ff56ba67d"}
Nov 02 22:20:13.720583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.715299    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:13.720583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.715385    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:13.733639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.733587    4929 generic.go:356] "Generic (PLEG): container finished" podID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerID="450f7e31105a3c9c69762f3f9520933f24c027ecf0519e987fc71ac8c11e4803" exitCode=0
Nov 02 22:20:13.734111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.734080    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" event={"ID":"f89c80cb-2ad9-4d92-8e7e-256cd29cab28","Type":"ContainerDied","Data":"450f7e31105a3c9c69762f3f9520933f24c027ecf0519e987fc71ac8c11e4803"}
Nov 02 22:20:13.815093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.815056    4929 generic.go:356] "Generic (PLEG): container finished" podID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerID="88b5f3ccb9203c86849451b823d7452f535268153df846770025bfa5ac811031" exitCode=0
Nov 02 22:20:13.815810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.815780    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" event={"ID":"6bda60b1-00f8-4dc5-af5e-9459abcfe271","Type":"ContainerDied","Data":"88b5f3ccb9203c86849451b823d7452f535268153df846770025bfa5ac811031"}
Nov 02 22:20:13.867576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.865663    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.898759    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.898842    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.901139    4929 generic.go:356] "Generic (PLEG): container finished" podID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerID="607b6ae88b9c6178191dd0f5001967b857d8795cf2864cecdb0334da4a6d444d" exitCode=0
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.920607    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" event={"ID":"915f1229-f294-438f-aeef-f3ee3d57a67c","Type":"ContainerDied","Data":"607b6ae88b9c6178191dd0f5001967b857d8795cf2864cecdb0334da4a6d444d"}
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.920787    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:13.939606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:13.920830    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:14.263626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.225370    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w" podUID="c137d510-9d3e-4e3b-94ec-8daf581918f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:14.263626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.232851    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:20:14.297702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.297654    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:14.628554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.628179    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:20:14.695590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.671431    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:14.695590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.671559    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:14.695590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.672118    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:14.695590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.672630    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:14.695590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:14.691956    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:20:15.027452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.027394    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:20:15.045555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.032736    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:15.055679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.055616    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:15.311904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.311863    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360409    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360512    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360589    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360643    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360686    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360737    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360789    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360820    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.360879    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361017    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361045    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361163    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361205    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361306    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361389    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361449    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361476    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\") pod \"165850e3-d69d-414d-906a-67cb59e15405\" (UID: \"165850e3-d69d-414d-906a-67cb59e15405\") "
Nov 02 22:20:15.388493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.361519    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\") pod \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\" (UID: \"6bda60b1-00f8-4dc5-af5e-9459abcfe271\") "
Nov 02 22:20:15.392591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.390272    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 22:20:15.398967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.398372    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.414478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.414430    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:15.431603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.414740    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:15.433816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.418723    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.442556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.421596    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 22:20:15.539814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.539070    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.539814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.539488    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.548689    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.568379    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.568814    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.569085    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.571640    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.577076    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.577215    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.577285    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.577528    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.581765    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.581819    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.581897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.581855    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.582907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.581969    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.582907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.582057    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\") pod \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\" (UID: \"f89c80cb-2ad9-4d92-8e7e-256cd29cab28\") "
Nov 02 22:20:15.592572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.583164    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.592842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.592815    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.592952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.592939    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593023    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165850e3-d69d-414d-906a-67cb59e15405-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593104    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593182    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593266    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593343    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.593428 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.593418    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.644688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.644643    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.645258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.645234    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.651958    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.652172    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.653002    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.654328    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w"
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.730901    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.730942    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.731149    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy" containerID="cri-o://53af4bc8a6502bcc24df705fabc786c04f62826cfa4c389d7a0fc2544829dc22" gracePeriod=30
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.730640    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.732830    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" containerID="cri-o://788cab08fe2e6575f191cbd75a1248548876a8622ed8882b4622eae8e6c735ce" gracePeriod=30
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.733648    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:15.765558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.749376    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.801755    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.812080    4929 patch_prober.go:29] interesting pod/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.812134    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.818969    4929 generic.go:356] "Generic (PLEG): container finished" podID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerID="ff85df32a437a53e6c13768b95052a204e45bb140105090532e9d6b19298da0d" exitCode=0
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.819129    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" event={"ID":"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c","Type":"ContainerDied","Data":"ff85df32a437a53e6c13768b95052a204e45bb140105090532e9d6b19298da0d"}
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823610    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823638    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823653    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823667    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823682    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823697    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.823713    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.825500    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.825556    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.830106    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7" event={"ID":"6bda60b1-00f8-4dc5-af5e-9459abcfe271","Type":"ContainerDied","Data":"81051875906f7893032527192e030b1801356c8dc2b77b0a267835904dae5088"}
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.830178    4929 scope.go:122] "RemoveContainer" containerID="88b5f3ccb9203c86849451b823d7452f535268153df846770025bfa5ac811031"
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.830374    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.833817    4929 generic.go:356] "Generic (PLEG): container finished" podID="d53db915-22df-430d-9c2a-257a68c889a0" containerID="4e55ee743b68e1567afbc5c4b413e9ecb3f0736128fdd58fc030eacf0eddbe5e" exitCode=0
Nov 02 22:20:15.872645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.833907    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" event={"ID":"d53db915-22df-430d-9c2a-257a68c889a0","Type":"ContainerDied","Data":"4e55ee743b68e1567afbc5c4b413e9ecb3f0736128fdd58fc030eacf0eddbe5e"}
Nov 02 22:20:15.878685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.878635    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:15.878934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.878893    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:15.899356 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.887742    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:20:15.899356 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.887972    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" containerID="cri-o://49a9528177939c03ffdce223654aa35eb72c6681be4483126ee5b5ed7aaef752" gracePeriod=30
Nov 02 22:20:15.910414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.910343    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:15.910792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.910753    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:15.913885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.913129    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:15.916306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.915349    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79" event={"ID":"165850e3-d69d-414d-906a-67cb59e15405","Type":"ContainerDied","Data":"8038784e3817283f865178a19b854d2c59c08526922c9dd76be7905ccaa26689"}
Nov 02 22:20:15.916306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.915486    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"
Nov 02 22:20:15.923102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.918891    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "kube-api-access-7grm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:15.923102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.918980    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "kube-api-access-7mgp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:15.923102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.920903    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk" event={"ID":"f89c80cb-2ad9-4d92-8e7e-256cd29cab28","Type":"ContainerDied","Data":"3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d"}
Nov 02 22:20:15.923102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.922066    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.924600    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.924685    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.924984    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925032    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925075    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925143    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925179    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925209    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.925336    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\") pod \"915f1229-f294-438f-aeef-f3ee3d57a67c\" (UID: \"915f1229-f294-438f-aeef-f3ee3d57a67c\") "
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926240    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grm2\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-kube-api-access-7grm2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926266    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926282    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165850e3-d69d-414d-906a-67cb59e15405-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926297    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgp4\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-kube-api-access-7mgp4\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926312    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:15.926965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.926587    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.928767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.928169    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.928767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.928472    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:15.929820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.929081    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:15.929820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:15.929271    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:16.018895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.000712    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:16.018895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.000861    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "kube-api-access-4k2wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.031296 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.031248    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 22:20:16.119583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.061381    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/165850e3-d69d-414d-906a-67cb59e15405-kube-api-access-4k2wp\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.230490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.230439    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.230866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.230811    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.230971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.230958    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.231055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.231044    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.231131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.231121    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.231209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.231198    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915f1229-f294-438f-aeef-f3ee3d57a67c-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.231307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.157798    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.273592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.265991    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:16.274089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.274021    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "kube-api-access-t6jnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.283021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.282944    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.343078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.331004    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy" pod "915f1229-f294-438f-aeef-f3ee3d57a67c" (UID: "915f1229-f294-438f-aeef-f3ee3d57a67c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:16.343078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.331640    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy" pod "165850e3-d69d-414d-906a-67cb59e15405" (UID: "165850e3-d69d-414d-906a-67cb59e15405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:16.343078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.333900    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy" pod "f89c80cb-2ad9-4d92-8e7e-256cd29cab28" (UID: "f89c80cb-2ad9-4d92-8e7e-256cd29cab28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.346733    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy" pod "6bda60b1-00f8-4dc5-af5e-9459abcfe271" (UID: "6bda60b1-00f8-4dc5-af5e-9459abcfe271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348283    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348309    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-kube-api-access-t6jnn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348325    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348338    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348351    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda60b1-00f8-4dc5-af5e-9459abcfe271-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348364    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89c80cb-2ad9-4d92-8e7e-256cd29cab28-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348377    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915f1229-f294-438f-aeef-f3ee3d57a67c-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.349119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.348390    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165850e3-d69d-414d-906a-67cb59e15405-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.417661 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.416171    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:20:16.421689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.421652    4929 scope.go:122] "RemoveContainer" containerID="2afa3e4b077949cf03bc8b98e06ec0c992af7de15847f59e0d50d37ff56ba67d"
Nov 02 22:20:16.458704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.458666    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:20:16.461585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.461558    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:20:16.530685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.529999    4929 scope.go:122] "RemoveContainer" containerID="450f7e31105a3c9c69762f3f9520933f24c027ecf0519e987fc71ac8c11e4803"
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.582297    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.582398    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:16.582524    4929 logging.go:55] [core] [Channel #2505 SubChannel #2507]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: operation was canceled"
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.583922    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\" (UniqueName: \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\") pod \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") "
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.584141    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\") pod \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\" (UID: \"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c\") "
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.587629    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\") pod \"d53db915-22df-430d-9c2a-257a68c889a0\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") "
Nov 02 22:20:16.607861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.587880    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\" (UniqueName: \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\") pod \"d53db915-22df-430d-9c2a-257a68c889a0\" (UID: \"d53db915-22df-430d-9c2a-257a68c889a0\") "
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.611387    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.611445    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.611896    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:16.612008    4929 logging.go:55] [core] [Channel #2506 SubChannel #2508]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.43:50052", ServerName: "10.133.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.43:50052: operation was canceled"
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.613664    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.613713    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.614097    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume" pod "35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" (UID: "35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:16.616219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.614495    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume" pod "d53db915-22df-430d-9c2a-257a68c889a0" (UID: "d53db915-22df-430d-9c2a-257a68c889a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:16.618813 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.618621    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-62mt7"]
Nov 02 22:20:16.619254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.619211    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp" pod "35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" (UID: "35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c"). InnerVolumeSpecName "kube-api-access-2mfqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.620930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.620904    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd" pod "d53db915-22df-430d-9c2a-257a68c889a0" (UID: "d53db915-22df-430d-9c2a-257a68c889a0"). InnerVolumeSpecName "kube-api-access-kb5cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:16.728695 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.726681    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:20:16.730481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.730378    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfqp\" (UniqueName: \"kubernetes.io/projected/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-kube-api-access-2mfqp\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.730481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.730409    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.730481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.730424    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53db915-22df-430d-9c2a-257a68c889a0-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.730481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.730440    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5cd\" (UniqueName: \"kubernetes.io/projected/d53db915-22df-430d-9c2a-257a68c889a0-kube-api-access-kb5cd\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:16.776618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.751791    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-9vg79"]
Nov 02 22:20:16.803603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.799718    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.823931    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165850e3-d69d-414d-906a-67cb59e15405" path="/var/lib/kubelet/pods/165850e3-d69d-414d-906a-67cb59e15405/volumes"
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.824908    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" path="/var/lib/kubelet/pods/6bda60b1-00f8-4dc5-af5e-9459abcfe271/volumes"
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.826429    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-c7kvk"]
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.827815    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.828885    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:20:16.838568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.830852    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:20:16.911361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.906927    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 02 22:20:16.911361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.906987    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 02 22:20:16.911717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.911390    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913387    4929 cpu_manager.go:414] "containerMap: removing container" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913412    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913453    4929 cpu_manager.go:414] "containerMap: removing container" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913462    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913511    4929 cpu_manager.go:414] "containerMap: removing container" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913519    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913557    4929 cpu_manager.go:414] "containerMap: removing container" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913565    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913618    4929 cpu_manager.go:414] "containerMap: removing container" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913637    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913667    4929 cpu_manager.go:414] "containerMap: removing container" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.913675    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914120    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="165850e3-d69d-414d-906a-67cb59e15405" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914140    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53db915-22df-430d-9c2a-257a68c889a0" containerName="payload-pre-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914162    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914185    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" containerName="payload-processing"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914204    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bda60b1-00f8-4dc5-af5e-9459abcfe271" containerName="istio-proxy"
Nov 02 22:20:16.917351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.914224    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" containerName="istio-proxy"
Nov 02 22:20:16.923185 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.923143    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:16.930916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.930817    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:20:16.931969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:16.931878    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a54217-serving-cert\""
Nov 02 22:20:17.010676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.008976    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.010676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.009401    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtbv\" (UniqueName: \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.010676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.009440    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.010676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.009525    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.012349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.011688    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"
Nov 02 22:20:17.012349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.011782    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p" event={"ID":"35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c","Type":"ContainerDied","Data":"a4daedc8aad0a68b22aea861d70a0ba084ee265a615cbde4edecfde81bc45482"}
Nov 02 22:20:17.012349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.011840    4929 scope.go:122] "RemoveContainer" containerID="ff85df32a437a53e6c13768b95052a204e45bb140105090532e9d6b19298da0d"
Nov 02 22:20:17.020126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.019950    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4" event={"ID":"915f1229-f294-438f-aeef-f3ee3d57a67c","Type":"ContainerDied","Data":"ba7dafccf41478dfac9c31a2a109df6dea12b78e0da5df1e3c6d5c938e0de957"}
Nov 02 22:20:17.020126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.020114    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"
Nov 02 22:20:17.024380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.024356    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7" event={"ID":"d53db915-22df-430d-9c2a-257a68c889a0","Type":"ContainerDied","Data":"67e3415b38d5ae24fefe49201a80066a3b19cd71c6dc1cced8be17e7070bee3f"}
Nov 02 22:20:17.024497 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.024459    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"
Nov 02 22:20:17.027672 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:17.027019    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165850e3_d69d_414d_906a_67cb59e15405.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c80cb_2ad9_4d92_8e7e_256cd29cab28.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b0b6c3_a2c2_4c96_b223_4b3dd8aee51c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c80cb_2ad9_4d92_8e7e_256cd29cab28.slice/crio-3a47acc5b0d5af310eb371bacddb61a02c509ad985372fb063c8f15aae78520d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165850e3_d69d_414d_906a_67cb59e15405.slice/crio-8038784e3817283f865178a19b854d2c59c08526922c9dd76be7905ccaa26689\": RecentStats: unable to find data in memory cache]"
Nov 02 22:20:17.106561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.095029    4929 scope.go:122] "RemoveContainer" containerID="607b6ae88b9c6178191dd0f5001967b857d8795cf2864cecdb0334da4a6d444d"
Nov 02 22:20:17.201596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.171010    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtbv\" (UniqueName: \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.201596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.171061    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.201596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.171158    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.201596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.171230    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.201596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.187804    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.332219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.332143    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:20:17.347199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.347161    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.368487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.353702    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.373072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.373045    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1d78c-cf59c946c-9w86p"]
Nov 02 22:20:17.377319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.377259    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtbv\" (UniqueName: \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\") pod \"maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.394316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.394287    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:20:17.404089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.403942    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-6n4d4"]
Nov 02 22:20:17.410570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.410456    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:20:17.410943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.410921    4929 scope.go:122] "RemoveContainer" containerID="4e55ee743b68e1567afbc5c4b413e9ecb3f0736128fdd58fc030eacf0eddbe5e"
Nov 02 22:20:17.416468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.416426    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1d78c-6ff74fb6c5-pn5q7"]
Nov 02 22:20:17.575664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.575619    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 22:20:17.575871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.575681    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 22:20:17.610799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.610754    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:17.939610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.933461    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:17.939610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.935143    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:17.939610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:17.936906    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:20:18.059084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.059054    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:20:18.059683 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:18.059656    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1310f3a9_262e_48fd_b6bf_703a20a036d3.slice/crio-a7022ceb7d79d7ff79ec3bbe13456b8faff211db841ee5b62bb423f40f68f28e WatchSource:0}: Error finding container a7022ceb7d79d7ff79ec3bbe13456b8faff211db841ee5b62bb423f40f68f28e: Status 404 returned error can't find the container with id a7022ceb7d79d7ff79ec3bbe13456b8faff211db841ee5b62bb423f40f68f28e
Nov 02 22:20:18.060852 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.060832    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:20:18.200922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.200838    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:18.200922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.200896    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:18.421828 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.421794    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:18.422020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.421869    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:18.578742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.578666    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:20:18.580335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.580310    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:20:18.784081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.776481    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:18.811962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.811922    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c" path="/var/lib/kubelet/pods/35b0b6c3-a2c2-4c96-b223-4b3dd8aee51c/volumes"
Nov 02 22:20:18.812621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.812598    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915f1229-f294-438f-aeef-f3ee3d57a67c" path="/var/lib/kubelet/pods/915f1229-f294-438f-aeef-f3ee3d57a67c/volumes"
Nov 02 22:20:18.813134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.813114    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53db915-22df-430d-9c2a-257a68c889a0" path="/var/lib/kubelet/pods/d53db915-22df-430d-9c2a-257a68c889a0/volumes"
Nov 02 22:20:18.814040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.814019    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89c80cb-2ad9-4d92-8e7e-256cd29cab28" path="/var/lib/kubelet/pods/f89c80cb-2ad9-4d92-8e7e-256cd29cab28/volumes"
Nov 02 22:20:18.814481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:18.814463    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:19.068590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.068509    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" event={"ID":"1310f3a9-262e-48fd-b6bf-703a20a036d3","Type":"ContainerStarted","Data":"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"}
Nov 02 22:20:19.069084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.069049    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:19.069235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.069108    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" event={"ID":"1310f3a9-262e-48fd-b6bf-703a20a036d3","Type":"ContainerStarted","Data":"a7022ceb7d79d7ff79ec3bbe13456b8faff211db841ee5b62bb423f40f68f28e"}
Nov 02 22:20:19.080458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.080427    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:20:19.087930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.087872    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" podStartSLOduration=2.569417322 podStartE2EDuration="3.087854124s" totalImagesPullingTime="518.436802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:19.085258578 +0000 UTC m=+1356.393349382" watchObservedRunningTime="2026-11-02 22:20:19.087854124 +0000 UTC m=+1356.395944960"
Nov 02 22:20:19.892948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:19.892908    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:20:20.682031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.681974    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:20:20.682743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.682708    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" containerID="cri-o://5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963" gracePeriod=30
Nov 02 22:20:20.706756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.705558    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:20:20.706756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.705917    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" containerID="cri-o://1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7" gracePeriod=30
Nov 02 22:20:20.715244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.715211    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:20:20.715604 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:20.715578    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" containerID="cri-o://1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d" gracePeriod=30
Nov 02 22:20:21.152446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.152410    4929 generic.go:356] "Generic (PLEG): container finished" podID="3f131e84-9602-4858-871f-7547019f4e96" containerID="53af4bc8a6502bcc24df705fabc786c04f62826cfa4c389d7a0fc2544829dc22" exitCode=0
Nov 02 22:20:21.153333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.152890    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" event={"ID":"3f131e84-9602-4858-871f-7547019f4e96","Type":"ContainerDied","Data":"53af4bc8a6502bcc24df705fabc786c04f62826cfa4c389d7a0fc2544829dc22"}
Nov 02 22:20:21.162009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.161974    4929 generic.go:356] "Generic (PLEG): container finished" podID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerID="788cab08fe2e6575f191cbd75a1248548876a8622ed8882b4622eae8e6c735ce" exitCode=0
Nov 02 22:20:21.165603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.162482    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" event={"ID":"1206c0ba-96a6-44b1-b933-df3f858bed5a","Type":"ContainerDied","Data":"788cab08fe2e6575f191cbd75a1248548876a8622ed8882b4622eae8e6c735ce"}
Nov 02 22:20:21.171595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.170767    4929 generic.go:356] "Generic (PLEG): container finished" podID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerID="49a9528177939c03ffdce223654aa35eb72c6681be4483126ee5b5ed7aaef752" exitCode=0
Nov 02 22:20:21.171595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.170856    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" event={"ID":"19b95825-ce66-4f77-8eae-03c35fdd0e4d","Type":"ContainerDied","Data":"49a9528177939c03ffdce223654aa35eb72c6681be4483126ee5b5ed7aaef752"}
Nov 02 22:20:21.568599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.568256    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:20:21.660644 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.660601    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.660946 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.660923    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.661090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.661074    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.661493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.661470    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.661680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.661665    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.661805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.661792    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.661933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.661920    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.662517 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.662496    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.669754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.669688    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\") pod \"1206c0ba-96a6-44b1-b933-df3f858bed5a\" (UID: \"1206c0ba-96a6-44b1-b933-df3f858bed5a\") "
Nov 02 22:20:21.681922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.681874    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:21.686654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.686615    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:21.689470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.689438    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:21.689868 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.689848    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:21.692798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.692762    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:21.775670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.775551    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206c0ba-96a6-44b1-b933-df3f858bed5a-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:21.775670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.775592    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:21.775670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.775609    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:21.775670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.775623    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:21.775670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.775636    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:21.894992 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.894934    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "kube-api-access-xztf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:21.896143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.896091    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:21.896803 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.896759    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:21.897600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.897371    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo" pod "1206c0ba-96a6-44b1-b933-df3f858bed5a" (UID: "1206c0ba-96a6-44b1-b933-df3f858bed5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:21.953217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.953186    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:20:21.991671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.991635    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:21.991948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.991931    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:21.992395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.992377    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.992294    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.993220    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.993263    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.993319    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.993848    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.993920    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.994014    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\") pod \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\" (UID: \"19b95825-ce66-4f77-8eae-03c35fdd0e4d\") "
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.994236    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.994759    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995299    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995324    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995339    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995352    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206c0ba-96a6-44b1-b933-df3f858bed5a-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995351    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995367    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztf6\" (UniqueName: \"kubernetes.io/projected/1206c0ba-96a6-44b1-b933-df3f858bed5a-kube-api-access-xztf6\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995383    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995397    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.995422    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.997551    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "kube-api-access-r5whr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.997600    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.997650    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.997799    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:22.031567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:21.999224    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.065737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.065669    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket" pod "19b95825-ce66-4f77-8eae-03c35fdd0e4d" (UID: "19b95825-ce66-4f77-8eae-03c35fdd0e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098070    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098108    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098123    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098137    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5whr\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-kube-api-access-r5whr\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098151    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.098516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.098164    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b95825-ce66-4f77-8eae-03c35fdd0e4d-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.127980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.127946    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.200826    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.200898    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201032    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201079    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201143    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201183    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201327    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201411    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201503    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.201947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.201608    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\") pod \"3f131e84-9602-4858-871f-7547019f4e96\" (UID: \"3f131e84-9602-4858-871f-7547019f4e96\") "
Nov 02 22:20:22.203918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.203751    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.204395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.204100    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.211503 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.211463    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.213361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.213333    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.217145 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.217120    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.217320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.217302    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.217466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.217450    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.217952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.217927    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:22.218334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.218312    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.236721    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.303292    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.281637    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.281486    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb" event={"ID":"19b95825-ce66-4f77-8eae-03c35fdd0e4d","Type":"ContainerDied","Data":"2f28407d33ecf030020da2b57708c8f2716e7db251b6dca093cdf44ec313c145"}
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.309498    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h" event={"ID":"3f131e84-9602-4858-871f-7547019f4e96","Type":"ContainerDied","Data":"ac9db5d1f5c61256cfd2196ee2353693c0d761629ba883f3eea96ae6a2e0c3cf"}
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.307620    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"
Nov 02 22:20:22.310226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.309645    4929 scope.go:122] "RemoveContainer" containerID="49a9528177939c03ffdce223654aa35eb72c6681be4483126ee5b5ed7aaef752"
Nov 02 22:20:22.317730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.317694    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b" event={"ID":"1206c0ba-96a6-44b1-b933-df3f858bed5a","Type":"ContainerDied","Data":"56fe46bbf5a6b06003354a8bc2e7e4256bf62effa37b73660c4b963e4e2cab81"}
Nov 02 22:20:22.324602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.320275    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"
Nov 02 22:20:22.332178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.332151    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f131e84-9602-4858-871f-7547019f4e96-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.332353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.332336    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f131e84-9602-4858-871f-7547019f4e96-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.332478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.332462    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.334209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.334187    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f131e84-9602-4858-871f-7547019f4e96-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.375651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.359803    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz" pod "3f131e84-9602-4858-871f-7547019f4e96" (UID: "3f131e84-9602-4858-871f-7547019f4e96"). InnerVolumeSpecName "kube-api-access-4dvnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:22.389389 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.389360    4929 scope.go:122] "RemoveContainer" containerID="53af4bc8a6502bcc24df705fabc786c04f62826cfa4c389d7a0fc2544829dc22"
Nov 02 22:20:22.400788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.400384    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:20:22.404623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.404596    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-6lmvb"]
Nov 02 22:20:22.405832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.405812    4929 scope.go:122] "RemoveContainer" containerID="788cab08fe2e6575f191cbd75a1248548876a8622ed8882b4622eae8e6c735ce"
Nov 02 22:20:22.420223 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.420197    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:20:22.424276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.424252    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-wtm7b"]
Nov 02 22:20:22.439582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.438977    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dvnz\" (UniqueName: \"kubernetes.io/projected/3f131e84-9602-4858-871f-7547019f4e96-kube-api-access-4dvnz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:22.760936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.760892    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:20:22.762171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.762139    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-flv2h"]
Nov 02 22:20:22.817565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.817516    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" path="/var/lib/kubelet/pods/1206c0ba-96a6-44b1-b933-df3f858bed5a/volumes"
Nov 02 22:20:22.818555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.818510    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" path="/var/lib/kubelet/pods/19b95825-ce66-4f77-8eae-03c35fdd0e4d/volumes"
Nov 02 22:20:22.819305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.819285    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f131e84-9602-4858-871f-7547019f4e96" path="/var/lib/kubelet/pods/3f131e84-9602-4858-871f-7547019f4e96/volumes"
Nov 02 22:20:22.874148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.873793    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:22.874148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:22.873866    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:24.215853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:24.215813    4929 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:20:24.216302 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:24.215882    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:20:25.111518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:25.111489    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:20:26.193488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.193455    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:20:26.222329 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.220177    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266440    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266518    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266579    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266693    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266731    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266785    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266833    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266905    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.266962    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267080    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267128    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267188    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267226    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267341    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267372    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267439    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267468    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\") pod \"35494137-6f30-43ae-a2e1-f0ba686330d8\" (UID: \"35494137-6f30-43ae-a2e1-f0ba686330d8\") "
Nov 02 22:20:26.267616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.267513    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\") pod \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\" (UID: \"7a215371-3f8c-43f5-b22d-ab9d0c6895c3\") "
Nov 02 22:20:26.275381 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.275327    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:26.276014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.275989    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:26.280077 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.279786    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:26.280406 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.280259    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.282011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.281819    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.288353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.287860    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "kube-api-access-tvcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.288607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.288571    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.288968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.288855    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.289188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.289051    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.289407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.289285    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.289614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.289481    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.290747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.290718    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.290943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.290918    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.294111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.294073    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:26.302867 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.302691    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.308134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.308007    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.308134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.308087    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2" pod "7a215371-3f8c-43f5-b22d-ab9d0c6895c3" (UID: "7a215371-3f8c-43f5-b22d-ab9d0c6895c3"). InnerVolumeSpecName "kube-api-access-hcjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.309914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.309881    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy" pod "35494137-6f30-43ae-a2e1-f0ba686330d8" (UID: "35494137-6f30-43ae-a2e1-f0ba686330d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.372420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.372268    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:20:26.376235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376206    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376234    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376244    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376254    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcz5\" (UniqueName: \"kubernetes.io/projected/35494137-6f30-43ae-a2e1-f0ba686330d8-kube-api-access-tvcz5\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376264    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376272    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjp2\" (UniqueName: \"kubernetes.io/projected/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-kube-api-access-hcjp2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376281    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35494137-6f30-43ae-a2e1-f0ba686330d8-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376290    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376298    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376306    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376317    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376325    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376333    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376342    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376351    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35494137-6f30-43ae-a2e1-f0ba686330d8-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376359    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376367    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.376385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.376376    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a215371-3f8c-43f5-b22d-ab9d0c6895c3-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.408519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.408484    4929 generic.go:356] "Generic (PLEG): container finished" podID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerID="1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7" exitCode=0
Nov 02 22:20:26.408743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.408611    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" event={"ID":"7a215371-3f8c-43f5-b22d-ab9d0c6895c3","Type":"ContainerDied","Data":"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"}
Nov 02 22:20:26.408743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.408653    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d" event={"ID":"7a215371-3f8c-43f5-b22d-ab9d0c6895c3","Type":"ContainerDied","Data":"dd1a5cb1376e5ae737668bc2b2c7b3714f27b418f278ca673e8f7f50c72bf14c"}
Nov 02 22:20:26.408743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.408676    4929 scope.go:122] "RemoveContainer" containerID="1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"
Nov 02 22:20:26.408912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.408797    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"
Nov 02 22:20:26.413241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.413213    4929 generic.go:356] "Generic (PLEG): container finished" podID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerID="5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963" exitCode=0
Nov 02 22:20:26.413461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.413274    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" event={"ID":"35494137-6f30-43ae-a2e1-f0ba686330d8","Type":"ContainerDied","Data":"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"}
Nov 02 22:20:26.413591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.413313    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"
Nov 02 22:20:26.415588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.413510    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss" event={"ID":"35494137-6f30-43ae-a2e1-f0ba686330d8","Type":"ContainerDied","Data":"1ffa6aca429f0cd439e7fc837cc6fa1566eb227c363417e50dca59da4ddcad90"}
Nov 02 22:20:26.420125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.418428    4929 generic.go:356] "Generic (PLEG): container finished" podID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerID="1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d" exitCode=0
Nov 02 22:20:26.420125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.418465    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"
Nov 02 22:20:26.420125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.418489    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" event={"ID":"4bdde8b0-b059-4ba1-a0ad-41de69c3974e","Type":"ContainerDied","Data":"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"}
Nov 02 22:20:26.420125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.418554    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b" event={"ID":"4bdde8b0-b059-4ba1-a0ad-41de69c3974e","Type":"ContainerDied","Data":"7372fae050648cf41878c4f32570033bc58926ae7947b13ca2455b156fb2ea5f"}
Nov 02 22:20:26.431797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.431775    4929 scope.go:122] "RemoveContainer" containerID="1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"
Nov 02 22:20:26.432168 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:26.432134    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7\": container with ID starting with 1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7 not found: ID does not exist" containerID="1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"
Nov 02 22:20:26.432251 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.432180    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7"} err="failed to get container status \"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7\": rpc error: code = NotFound desc = could not find container \"1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7\": container with ID starting with 1bfc6ada6029173b5f6f6f2468ea4db71ed196b41d742386e89b83a5172866f7 not found: ID does not exist"
Nov 02 22:20:26.432251 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.432201    4929 scope.go:122] "RemoveContainer" containerID="5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"
Nov 02 22:20:26.477556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477514    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.477833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477745    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.477833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477783    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.477974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477870    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.477974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477939    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.478089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.477982    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.478460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.478147    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.478460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.478182    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.478460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.478314    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\") pod \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\" (UID: \"4bdde8b0-b059-4ba1-a0ad-41de69c3974e\") "
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.479762    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.480268    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.480484    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.480756    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.481073    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:20:26.482246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.481937    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:20:26.482452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.482421    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:20:26.484580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.482873    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:20:26.484580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.482989    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.485484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.485368    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r" pod "4bdde8b0-b059-4ba1-a0ad-41de69c3974e" (UID: "4bdde8b0-b059-4ba1-a0ad-41de69c3974e"). InnerVolumeSpecName "kube-api-access-n6l2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:20:26.487996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.487946    4929 scope.go:122] "RemoveContainer" containerID="5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"
Nov 02 22:20:26.488635 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:26.488475    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963\": container with ID starting with 5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963 not found: ID does not exist" containerID="5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"
Nov 02 22:20:26.488635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.488515    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963"} err="failed to get container status \"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963\": rpc error: code = NotFound desc = could not find container \"5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963\": container with ID starting with 5a36a65b60cdd6dce24d3a9f08bacccb234fca4e782cce67181bd71db984a963 not found: ID does not exist"
Nov 02 22:20:26.488635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.488557    4929 scope.go:122] "RemoveContainer" containerID="1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"
Nov 02 22:20:26.490759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.490743    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-lc89d"]
Nov 02 22:20:26.496334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.496301    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:20:26.504176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.501566    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-r7sss"]
Nov 02 22:20:26.509575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.509548    4929 scope.go:122] "RemoveContainer" containerID="1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"
Nov 02 22:20:26.510149 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:26.510126    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d\": container with ID starting with 1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d not found: ID does not exist" containerID="1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"
Nov 02 22:20:26.510357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.510158    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d"} err="failed to get container status \"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d\": rpc error: code = NotFound desc = could not find container \"1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d\": container with ID starting with 1723fd3a4fd0c0849a21271e7719aa5263435625c30af7bf9f8368ecd2e1da5d not found: ID does not exist"
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580625    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580662    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580672    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580680    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580690    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580699    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580708    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580716    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.580718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.580724    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6l2r\" (UniqueName: \"kubernetes.io/projected/4bdde8b0-b059-4ba1-a0ad-41de69c3974e-kube-api-access-n6l2r\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:20:26.772211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.772172    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:20:26.774026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.773997    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9wt7b"]
Nov 02 22:20:26.807292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.807258    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" path="/var/lib/kubelet/pods/35494137-6f30-43ae-a2e1-f0ba686330d8/volumes"
Nov 02 22:20:26.807881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.807865    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" path="/var/lib/kubelet/pods/4bdde8b0-b059-4ba1-a0ad-41de69c3974e/volumes"
Nov 02 22:20:26.808366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:26.808352    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" path="/var/lib/kubelet/pods/7a215371-3f8c-43f5-b22d-ab9d0c6895c3/volumes"
Nov 02 22:20:27.500390 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:27.500356    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:20:28.334576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.334524    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:20:28.335334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335318    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy"
Nov 02 22:20:28.335382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335336    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy"
Nov 02 22:20:28.335382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335346    4929 cpu_manager.go:414] "containerMap: removing container" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy"
Nov 02 22:20:28.335382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335351    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy"
Nov 02 22:20:28.335382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335378    4929 cpu_manager.go:414] "containerMap: removing container" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335384    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335407    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335412    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335429    4929 cpu_manager.go:414] "containerMap: removing container" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335434    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335440    4929 cpu_manager.go:414] "containerMap: removing container" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy"
Nov 02 22:20:28.335496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335445    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335574    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1206c0ba-96a6-44b1-b933-df3f858bed5a" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335583    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdde8b0-b059-4ba1-a0ad-41de69c3974e" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335594    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f131e84-9602-4858-871f-7547019f4e96" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335605    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a215371-3f8c-43f5-b22d-ab9d0c6895c3" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335613    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="35494137-6f30-43ae-a2e1-f0ba686330d8" containerName="istio-proxy"
Nov 02 22:20:28.335714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.335619    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b95825-ce66-4f77-8eae-03c35fdd0e4d" containerName="istio-proxy"
Nov 02 22:20:28.341216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.341195    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.348332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.348219    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:20:28.350086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.350056    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e49edd-serving-cert\""
Nov 02 22:20:28.410144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.410098    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.410338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.410234    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.410338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.410276    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.410338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.410301    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bw5s\" (UniqueName: \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.512648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.512603    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.513122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.512726    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.513122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.512789    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.513122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.512821    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw5s\" (UniqueName: \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.513122 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:28.512932    4929 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-serving-cert: secret "maas-api-e2e-inf-b-e49edd-serving-cert" not found
Nov 02 22:20:28.513122 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:28.513015    4929 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls podName:354a47a7-8241-4d2a-a09c-dc75aaa12cd5 nodeName:}" failed. No retries permitted until 2026-11-02 22:20:29.012993846 +0000 UTC m=+1366.321084648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls") pod "maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" (UID: "354a47a7-8241-4d2a-a09c-dc75aaa12cd5") : secret "maas-api-e2e-inf-b-e49edd-serving-cert" not found
Nov 02 22:20:28.513876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.513846    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.518635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.515822    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:28.527096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:28.527057    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bw5s\" (UniqueName: \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:29.021101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:29.021062    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:29.023718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:29.023689    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") pod \"maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:29.269466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:29.269416    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:29.568710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:29.568672    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:20:29.569984 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:29.569949    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354a47a7_8241_4d2a_a09c_dc75aaa12cd5.slice/crio-08e4f485d2cac4d8de659ccef311783a761be1599c8700cd97be61108d9bdbe8 WatchSource:0}: Error finding container 08e4f485d2cac4d8de659ccef311783a761be1599c8700cd97be61108d9bdbe8: Status 404 returned error can't find the container with id 08e4f485d2cac4d8de659ccef311783a761be1599c8700cd97be61108d9bdbe8
Nov 02 22:20:30.453112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.453007    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:20:30.472728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.469859    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.472728 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.472203    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:20:30.479292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.479253    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" event={"ID":"354a47a7-8241-4d2a-a09c-dc75aaa12cd5","Type":"ContainerStarted","Data":"47c44f393eee673498bcbdf6e2e5784f31bf27579e0fbc1eeb342b88c6ddc3e2"}
Nov 02 22:20:30.479426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.479321    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" event={"ID":"354a47a7-8241-4d2a-a09c-dc75aaa12cd5","Type":"ContainerStarted","Data":"08e4f485d2cac4d8de659ccef311783a761be1599c8700cd97be61108d9bdbe8"}
Nov 02 22:20:30.480268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.480237    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:30.480446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.480427    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d28bd1-openshift-default-dockercfg-9p2fd\""
Nov 02 22:20:30.516643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.516583    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" podStartSLOduration=1.97309527 podStartE2EDuration="2.516564153s" totalImagesPullingTime="543.468883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:30.513197073 +0000 UTC m=+1367.821287889" watchObservedRunningTime="2026-11-02 22:20:30.516564153 +0000 UTC m=+1367.824654957"
Nov 02 22:20:30.579468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.579429    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.579901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.579486    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.579901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.579772    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db276\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.579901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.579831    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.580018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.579981    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.580117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.580089    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.580173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.580131    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.580232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.580175    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.580523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.580493    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682641    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db276\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682699    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682770    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682835    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682855    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.682880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682869    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.682935    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683037    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683075    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683302    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683378    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683428    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683656    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.683959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.683930    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.685626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.685597    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.685782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.685767    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.690494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.690461    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db276\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.691712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.691681    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:30.812162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:30.811828    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:31.216109 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:31.216070    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0e783b_b508_4204_bd25_c411160af26e.slice/crio-2b562a69a641bb8b37b9b2f61904496cd0e0b84d88422abd8e3c6ff7206eb7e6 WatchSource:0}: Error finding container 2b562a69a641bb8b37b9b2f61904496cd0e0b84d88422abd8e3c6ff7206eb7e6: Status 404 returned error can't find the container with id 2b562a69a641bb8b37b9b2f61904496cd0e0b84d88422abd8e3c6ff7206eb7e6
Nov 02 22:20:31.217721 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.217684    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:20:31.219316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.219270    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:31.219449 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.219352    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:31.219449 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.219387    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:31.493338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.493228    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerStarted","Data":"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"}
Nov 02 22:20:31.493338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.493313    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerStarted","Data":"2b562a69a641bb8b37b9b2f61904496cd0e0b84d88422abd8e3c6ff7206eb7e6"}
Nov 02 22:20:31.513306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.513242    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podStartSLOduration=1.513224985 podStartE2EDuration="1.513224985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:31.511395299 +0000 UTC m=+1368.819486106" watchObservedRunningTime="2026-11-02 22:20:31.513224985 +0000 UTC m=+1368.821315789"
Nov 02 22:20:31.813345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:31.813250    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:32.819714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:32.819621    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:33.513671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.513639    4929 generic.go:356] "Generic (PLEG): container finished" podID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerID="a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16" exitCode=0
Nov 02 22:20:33.513943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.513712    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerDied","Data":"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"}
Nov 02 22:20:33.514030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.514011    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:33.514482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.514456    4929 scope.go:122] "RemoveContainer" containerID="a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"
Nov 02 22:20:33.515435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.515100    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:20:33.516312 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.516280    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:33.516409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.516352    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:33.516409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:33.516400    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:34.523281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:34.523245    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerStarted","Data":"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"}
Nov 02 22:20:34.899724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:34.899688    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:20:34.900837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:34.900807    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:34.900976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:34.900886    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:35.073829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.073788    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:20:35.082801 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.082770    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:20:35.083256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.083236    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.092009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.091979    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d28bd1\""
Nov 02 22:20:35.182616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.179169    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d28bd1-dockercfg-vqb6s\""
Nov 02 22:20:35.284576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.262879    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t82v\" (UniqueName: \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.284576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.262997    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.364778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.364728    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.365155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.365133    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t82v\" (UniqueName: \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.408684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.408629    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.433667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.433258    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t82v\" (UniqueName: \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\") pod \"payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.719259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.719174    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:35.874207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.873635    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:20:35.900079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.900041    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:35.900259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:35.900158    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:36.033468 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:36.033437    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a9d4a4_eaed_4349_8b97_1f25f09c6444.slice/crio-fd34b6ecca5844b1d558a5fea2d9dfa09769fd2c72fb54978dc3382f222dc3b0 WatchSource:0}: Error finding container fd34b6ecca5844b1d558a5fea2d9dfa09769fd2c72fb54978dc3382f222dc3b0: Status 404 returned error can't find the container with id fd34b6ecca5844b1d558a5fea2d9dfa09769fd2c72fb54978dc3382f222dc3b0
Nov 02 22:20:36.058704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.058616    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:20:36.567298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.567248    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" event={"ID":"91a9d4a4-eaed-4349-8b97-1f25f09c6444","Type":"ContainerStarted","Data":"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"}
Nov 02 22:20:36.567508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.567357    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:36.567508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.567386    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" event={"ID":"91a9d4a4-eaed-4349-8b97-1f25f09c6444","Type":"ContainerStarted","Data":"fd34b6ecca5844b1d558a5fea2d9dfa09769fd2c72fb54978dc3382f222dc3b0"}
Nov 02 22:20:36.583482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.583424    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" podStartSLOduration=1.583409585 podStartE2EDuration="1.583409585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:36.582195547 +0000 UTC m=+1373.890286353" watchObservedRunningTime="2026-11-02 22:20:36.583409585 +0000 UTC m=+1373.891500445"
Nov 02 22:20:36.900376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.900335    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:36.900848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:36.900436    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:37.472212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.472173    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:20:37.477600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.477575    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.493366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.491711    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:20:37.507041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.506251    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a621f1-openshift-default-dockercfg-x96wf\""
Nov 02 22:20:37.517343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.517302    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.517633 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.517585    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.517786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.517720    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.517961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.517849    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.517961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.517914    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vfh\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.518195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.518163    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.518333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.518320    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.518484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.518473    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.518701 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.518683    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.532174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.528747    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:20:37.620143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620093    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620218    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620303    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620404    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620553    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620620    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620656    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620753    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.620842 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.620785    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vfh\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.621068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.621044    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.621146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.621106    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.621350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.621324    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.621889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.621862    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.622647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.622625    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.625938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.625914    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.629584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.629266    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.630912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.630884    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vfh\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.679058 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.679021    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\") pod \"e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") " pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.847439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.841118    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:37.902614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.902093    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:37.902614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:37.902168    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:38.333832 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:38.333798    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8637bbe2_fa8d_48ef_a592_b131944fed5f.slice/crio-81393df1e6cff0ca9ea1129e1684540bfe9a3b0988563074432e4f4b4a87305d WatchSource:0}: Error finding container 81393df1e6cff0ca9ea1129e1684540bfe9a3b0988563074432e4f4b4a87305d: Status 404 returned error can't find the container with id 81393df1e6cff0ca9ea1129e1684540bfe9a3b0988563074432e4f4b4a87305d
Nov 02 22:20:38.333971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.333861    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:20:38.362734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.362684    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:38.362895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.362792    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:38.362895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.362840    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:20:38.591749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.591715    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerStarted","Data":"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"}
Nov 02 22:20:38.592031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.591771    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerStarted","Data":"81393df1e6cff0ca9ea1129e1684540bfe9a3b0988563074432e4f4b4a87305d"}
Nov 02 22:20:38.612253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.612121    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podStartSLOduration=1.612099723 podStartE2EDuration="1.612099723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:38.611078112 +0000 UTC m=+1375.919168918" watchObservedRunningTime="2026-11-02 22:20:38.612099723 +0000 UTC m=+1375.920190532"
Nov 02 22:20:38.867620 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.855653    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:38.900228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.900178    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:38.900489 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:38.900263    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:39.867025 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:39.866988    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:39.899874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:39.899839    4929 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 22:20:39.900046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:39.899893    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 22:20:40.639487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:40.635703    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:40.639487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:40.636935    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:20:40.904502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:40.904425    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:20:40.905798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:40.905773    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:20:42.089157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.089117    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:20:42.095805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.095775    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.105726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.105699    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:20:42.118963 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.118932    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a621f1-dockercfg-mvwrf\""
Nov 02 22:20:42.173579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.173520    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxwm\" (UniqueName: \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.173773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.173743    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.190893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.190855    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a621f1\""
Nov 02 22:20:42.280101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.280054    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxwm\" (UniqueName: \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.280317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.280254    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.281399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.281368    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.368559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.366299    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxwm\" (UniqueName: \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\") pod \"payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.429603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.429375    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:42.727846 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:42.727801    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1525ec_7195_492c_a67c_fae208e3c824.slice/crio-d8dd2a97740e7c96ce24cad868ea1302d48b31d91ea5ea90a119055b681672ed WatchSource:0}: Error finding container d8dd2a97740e7c96ce24cad868ea1302d48b31d91ea5ea90a119055b681672ed: Status 404 returned error can't find the container with id d8dd2a97740e7c96ce24cad868ea1302d48b31d91ea5ea90a119055b681672ed
Nov 02 22:20:42.729041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:42.729005    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:20:43.697724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:43.697684    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" event={"ID":"7a1525ec-7195-492c-a67c-fae208e3c824","Type":"ContainerStarted","Data":"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"}
Nov 02 22:20:43.698150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:43.697745    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" event={"ID":"7a1525ec-7195-492c-a67c-fae208e3c824","Type":"ContainerStarted","Data":"d8dd2a97740e7c96ce24cad868ea1302d48b31d91ea5ea90a119055b681672ed"}
Nov 02 22:20:43.698150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:43.697780    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:43.714630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:43.714573    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" podStartSLOduration=1.714527719 podStartE2EDuration="1.714527719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:43.7128933 +0000 UTC m=+1381.020984114" watchObservedRunningTime="2026-11-02 22:20:43.714527719 +0000 UTC m=+1381.022618521"
Nov 02 22:20:47.583683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:47.583642    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:20:54.708984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:54.708886    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:20:55.987001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:55.986965    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:20:57.761556 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:20:57.760718    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:20:58.623577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.622048    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:20:58.629782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.629755    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.638898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.638865    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e49edd\"/\"openshift-service-ca.crt\""
Nov 02 22:20:58.639089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.639056    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e49edd\"/\"kube-root-ca.crt\""
Nov 02 22:20:58.639201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.639185    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e49edd\"/\"test-model-e49edd-kserve-self-signed-certs\""
Nov 02 22:20:58.639269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.639257    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e49edd\"/\"default-dockercfg-gmdb4\""
Nov 02 22:20:58.639863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.639833    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:20:58.716492 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716451    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.716684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716501    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgwm\" (UniqueName: \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.716684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716632    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.716796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716689    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.716796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716758    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.716864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.716816    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.835221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.835184    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.835831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.835806    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgwm\" (UniqueName: \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.836031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.836013    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.836196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.836181    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.836319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.836305    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.836472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.836457    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.838928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.838002    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.919039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.891123    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.919287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.896996    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.927553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.927502    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgwm\" (UniqueName: \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.976597 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.945958    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:58.984018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:58.983983    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\") pod \"test-model-e49edd-kserve-85c648dbb5-x7bjn\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") " pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:59.248475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:59.248380    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:20:59.531384 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:20:59.531343    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ae9d89a_7ecc_4671_a704_31d4a9dc0cda.slice/crio-47ad2bef8da7d9d42381d4d51917c9163a2a211c02f780948ac6d78a1d09ae94 WatchSource:0}: Error finding container 47ad2bef8da7d9d42381d4d51917c9163a2a211c02f780948ac6d78a1d09ae94: Status 404 returned error can't find the container with id 47ad2bef8da7d9d42381d4d51917c9163a2a211c02f780948ac6d78a1d09ae94
Nov 02 22:20:59.531747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:20:59.531723    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:21:00.004682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:00.004640    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" event={"ID":"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda","Type":"ContainerStarted","Data":"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"}
Nov 02 22:21:00.004682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:00.004689    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" event={"ID":"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda","Type":"ContainerStarted","Data":"47ad2bef8da7d9d42381d4d51917c9163a2a211c02f780948ac6d78a1d09ae94"}
Nov 02 22:21:00.023681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:00.023620    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" podStartSLOduration=2.02360614 podStartE2EDuration="2.02360614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:00.02134904 +0000 UTC m=+1397.329439846" watchObservedRunningTime="2026-11-02 22:21:00.02360614 +0000 UTC m=+1397.331696944"
Nov 02 22:21:00.248665 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:00.248628    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:21:06.694000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.693964    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:21:06.699200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.699178    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.710447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.710415    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:21:06.710746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.710619    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e49edd-openshift-default-dockercfg-qk94g\""
Nov 02 22:21:06.716363 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.716339    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:21:06.723630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.723603    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.768277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.768236    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:21:06.834629 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.834582    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.834829 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.834730    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.834902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.834835    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.834993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.834967    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.835732 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.835704    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.835853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.835797    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.835915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.835896    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.835968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.835955    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n6n\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.836011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.835982    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdms\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836020    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836068    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836234 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836217    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.836307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836294    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836369 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836324    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.836429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836367    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.836429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836394    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836547 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836452    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.836547 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.836495    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.937881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.937840    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.937892    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.937950    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.937993    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938014    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938128    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938229    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938260    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938312    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938341    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938341    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938348    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n6n\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938422    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdms\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938463    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938480    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938491    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938608    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938631    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938655    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938717    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938746    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938789    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938812    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.938938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.938860    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.939380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.939188    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.939380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.939204    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.939380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.939255    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.939570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.939473    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.940869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.940842    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.940990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.940874    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.941072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.941052    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.941475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.941452    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.946761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.946693    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n6n\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:06.946761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.946720    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdms\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.946936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.946830    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:06.946936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:06.946860    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:07.018321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.018280    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:07.059110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.059073    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:07.320154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.320119    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:21:07.320383 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:21:07.320361    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc797e29a_64d2_4f98_b179_ee4e332e00df.slice/crio-38d9d6ed89fe541b56d2317dbd9a4cb65262f589ec09c9589876a3a3f6e9c12a WatchSource:0}: Error finding container 38d9d6ed89fe541b56d2317dbd9a4cb65262f589ec09c9589876a3a3f6e9c12a: Status 404 returned error can't find the container with id 38d9d6ed89fe541b56d2317dbd9a4cb65262f589ec09c9589876a3a3f6e9c12a
Nov 02 22:21:07.323235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.323200    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:07.323321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.323269    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:07.323321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.323301    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:07.370272 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:21:07.370229    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c177830_557e_4f11_bc68_ba22db731fda.slice/crio-cc8fadeb93e8ba06b902f5974557a93bd79548b36a896f718b09162f35a86a72 WatchSource:0}: Error finding container cc8fadeb93e8ba06b902f5974557a93bd79548b36a896f718b09162f35a86a72: Status 404 returned error can't find the container with id cc8fadeb93e8ba06b902f5974557a93bd79548b36a896f718b09162f35a86a72
Nov 02 22:21:07.370450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.370420    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:21:07.373690 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.373652    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:07.373793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.373726    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:07.373793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:07.373781    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:08.154559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.142938    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" event={"ID":"c797e29a-64d2-4f98-b179-ee4e332e00df","Type":"ContainerStarted","Data":"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"}
Nov 02 22:21:08.154559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.143009    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" event={"ID":"c797e29a-64d2-4f98-b179-ee4e332e00df","Type":"ContainerStarted","Data":"38d9d6ed89fe541b56d2317dbd9a4cb65262f589ec09c9589876a3a3f6e9c12a"}
Nov 02 22:21:08.170745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.161871    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" event={"ID":"1c177830-557e-4f11-bc68-ba22db731fda","Type":"ContainerStarted","Data":"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"}
Nov 02 22:21:08.170745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.161952    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" event={"ID":"1c177830-557e-4f11-bc68-ba22db731fda","Type":"ContainerStarted","Data":"cc8fadeb93e8ba06b902f5974557a93bd79548b36a896f718b09162f35a86a72"}
Nov 02 22:21:08.192267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.191701    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podStartSLOduration=2.191683762 podStartE2EDuration="2.191683762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:08.17709395 +0000 UTC m=+1405.485184756" watchObservedRunningTime="2026-11-02 22:21:08.191683762 +0000 UTC m=+1405.499774567"
Nov 02 22:21:08.202230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:08.202176    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podStartSLOduration=2.202160721 podStartE2EDuration="2.202160721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:08.198911448 +0000 UTC m=+1405.507002253" watchObservedRunningTime="2026-11-02 22:21:08.202160721 +0000 UTC m=+1405.510251525"
Nov 02 22:21:09.018856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.018821    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:09.020080 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.020044    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:09.020220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.020101    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:09.074560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.073944    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:09.091931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.091305    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:09.091931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.091386    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:09.263561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.262616    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:21:09.299787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:09.299704    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:21:10.019674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:10.019634    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:10.019849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:10.019702    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:10.063245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:10.063207    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:10.063581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:10.063545    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:10.760127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:10.759916    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:21:11.019674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:11.019580    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:11.019674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:11.019633    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:11.090743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:11.090704    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:11.091024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:11.090993    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:12.019581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.019523    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:12.020208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.020162    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:12.063571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.059953    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:12.063571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.060025    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:12.327670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.327574    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:21:12.340760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.340725    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.346751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.346606    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a54217\"/\"test-model-a54217-kserve-self-signed-certs\""
Nov 02 22:21:12.346751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.346675    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a54217\"/\"openshift-service-ca.crt\""
Nov 02 22:21:12.346751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.346708    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a54217\"/\"default-dockercfg-tw2hz\""
Nov 02 22:21:12.347072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.346810    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a54217\"/\"kube-root-ca.crt\""
Nov 02 22:21:12.401255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.401159    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.401584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.401561    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhx9\" (UniqueName: \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.401807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.401795    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.401928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.401917    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.402071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.402045    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.402160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.402150    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.402753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.402562    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.511212    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.511296    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.511681    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.511712    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhx9\" (UniqueName: \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.512443    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.512710    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.515197    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.515683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.515364    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.521675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.516938    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.530123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.530079    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.530424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.530383    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.555679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.555594    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhx9\" (UniqueName: \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\") pod \"test-model-a54217-kserve-64c5b99579-z8dpn\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") " pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:12.744560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:12.739901    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:13.018997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.018825    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:13.018997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.018893    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:13.069556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.061470    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:13.102118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.101395    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:13.451703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.447570    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:21:13.464558 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:21:13.460944    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48e8a816_11c1_4d18_8487_87097f71cb93.slice/crio-626640f411050cf9456e4f1637fefcdcbb5b8a180a5f375361897db8ef8d76e2 WatchSource:0}: Error finding container 626640f411050cf9456e4f1637fefcdcbb5b8a180a5f375361897db8ef8d76e2: Status 404 returned error can't find the container with id 626640f411050cf9456e4f1637fefcdcbb5b8a180a5f375361897db8ef8d76e2
Nov 02 22:21:13.521353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:13.521323    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" event={"ID":"48e8a816-11c1-4d18-8487-87097f71cb93","Type":"ContainerStarted","Data":"626640f411050cf9456e4f1637fefcdcbb5b8a180a5f375361897db8ef8d76e2"}
Nov 02 22:21:14.019542 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.019504    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 22:21:14.019740 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.019590    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 22:21:14.077608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.077431    4929 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 22:21:14.077608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.077495    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 22:21:14.534063 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.534029    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" event={"ID":"48e8a816-11c1-4d18-8487-87097f71cb93","Type":"ContainerStarted","Data":"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"}
Nov 02 22:21:14.552622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.552550    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" podStartSLOduration=2.552522487 podStartE2EDuration="2.552522487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:14.549141408 +0000 UTC m=+1411.857232219" watchObservedRunningTime="2026-11-02 22:21:14.552522487 +0000 UTC m=+1411.860613291"
Nov 02 22:21:14.748686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:14.748651    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:15.024490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.024456    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:15.024974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.024946    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:15.026053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.026031    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:21:15.064243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.064211    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:15.064525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.064507    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:15.065345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:15.065322    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:21:18.197015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.196975    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:21:18.204667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.204639    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.210351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.210313    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a621f1\"/\"kube-root-ca.crt\""
Nov 02 22:21:18.211529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.211497    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:21:18.212624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.212603    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a621f1\"/\"openshift-service-ca.crt\""
Nov 02 22:21:18.216871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.215208    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a621f1\"/\"default-dockercfg-x6dl7\""
Nov 02 22:21:18.216871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.215275    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a621f1\"/\"test-model-a621f1-kserve-self-signed-certs\""
Nov 02 22:21:18.296928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.296888    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.297123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.296999    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.297123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.297041    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.297360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.297336    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.297407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.297390    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtwk\" (UniqueName: \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.297470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.297455    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399411    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399473    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtwk\" (UniqueName: \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399526    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399723    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399784    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.399961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.399815    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.400186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.400153    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.400186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.400169    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.400388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.400228    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.401997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.401970    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.402167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.402149    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.408165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.408132    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtwk\" (UniqueName: \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\") pod \"test-model-a621f1-kserve-5ff57648df-8grb6\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") " pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.545614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.539353    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:18.874618 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:21:18.874587    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7fe0f09_bf07_4dac_8e6d_6cfe589eb7e4.slice/crio-bede19b396bf6cde3a2151d3c1416c447feafea3db5d91cbcc7237f5846639e0 WatchSource:0}: Error finding container bede19b396bf6cde3a2151d3c1416c447feafea3db5d91cbcc7237f5846639e0: Status 404 returned error can't find the container with id bede19b396bf6cde3a2151d3c1416c447feafea3db5d91cbcc7237f5846639e0
Nov 02 22:21:18.875592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:18.875567    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:21:19.601118 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:19.601078    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" event={"ID":"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4","Type":"ContainerStarted","Data":"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"}
Nov 02 22:21:19.601669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:19.601130    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" event={"ID":"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4","Type":"ContainerStarted","Data":"bede19b396bf6cde3a2151d3c1416c447feafea3db5d91cbcc7237f5846639e0"}
Nov 02 22:21:19.621795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:19.621738    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" podStartSLOduration=1.621722557 podStartE2EDuration="1.621722557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:19.616727375 +0000 UTC m=+1416.924818181" watchObservedRunningTime="2026-11-02 22:21:19.621722557 +0000 UTC m=+1416.929813361"
Nov 02 22:21:20.540671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:20.540629    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:22.750921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:22.750884    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:22.779172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:22.779128    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:23.706052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:23.706016    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:21:27.625095 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:21:27.625061    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:21:28.561734 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:28.561688    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:28.600166 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:28.600127    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:28.730986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:28.730948    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:21:34.556743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:34.556699    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:21:34.557165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:34.556760    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:21:35.812955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.812922    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerDied","Data":"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"}
Nov 02 22:21:35.814086 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.814061    4929 scope.go:122] "RemoveContainer" containerID="5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"
Nov 02 22:21:35.814891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.812876    4929 generic.go:356] "Generic (PLEG): container finished" podID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerID="5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a" exitCode=0
Nov 02 22:21:35.816649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.816613    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:35.816766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.816701    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:35.816766 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:35.816754    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:36.830270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.830184    4929 generic.go:356] "Generic (PLEG): container finished" podID="3c0e783b-b508-4204-bd25-c411160af26e" containerID="2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895" exitCode=0
Nov 02 22:21:36.830270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.830253    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerDied","Data":"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"}
Nov 02 22:21:36.830973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.830949    4929 scope.go:122] "RemoveContainer" containerID="2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"
Nov 02 22:21:36.833437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.833399    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:36.833599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.833496    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:36.833684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.833600    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:21:36.833832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.833812    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerStarted","Data":"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"}
Nov 02 22:21:36.857729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.857272    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:21:36.858051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.858022    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:36.858211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:36.858186    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:37.889714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:37.889465    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:37.889714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:37.889577    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:37.900678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:37.900640    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerStarted","Data":"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"}
Nov 02 22:21:38.813105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:38.813069    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:21:38.813311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:38.813290    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 22:21:38.813358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:38.813336    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 22:21:38.857558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:38.854521    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:38.857558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:38.854610    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:39.695621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.695583    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:21:39.720311 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.720269    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:21:39.819252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.814089    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 22:21:39.819252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.814158    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 22:21:39.882569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.881665    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:39.882569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:39.881738    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:40.635683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:40.635644    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:21:40.814250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:40.814218    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 22:21:40.814771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:40.814286    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 22:21:40.857559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:40.851904    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:40.857559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:40.852002    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:41.813399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:41.813361    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 22:21:41.813668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:41.813437    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 22:21:41.856980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:41.856941    4929 patch_prober.go:29] interesting pod/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 22:21:41.858044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:41.857638    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 22:21:42.813300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:42.813268    4929 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 22:21:42.813491 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:42.813314    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 22:21:42.884566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:42.884144    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:21:42.893611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:42.893378    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:21:43.817687 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:43.817655    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:21:43.818169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:43.818156    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:21:43.818965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:43.818938    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:21:44.696032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:44.695985    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" podUID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" containerName="payload-processing" containerID="cri-o://47c155319b84e7189950cbf237692d80f35886fb70ba9ec47b027258f8171565" gracePeriod=25
Nov 02 22:21:44.720673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:44.720612    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" podUID="51d676b3-6802-4995-b9f0-44e42cbff2d3" containerName="payload-pre-processing" containerID="cri-o://32b74c28e341ffad413b260d84b9cd269dfa0a9901bc911d645625b3c30bece1" gracePeriod=25
Nov 02 22:21:45.027099 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.023344    4929 generic.go:356] "Generic (PLEG): container finished" podID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" containerID="47c155319b84e7189950cbf237692d80f35886fb70ba9ec47b027258f8171565" exitCode=0
Nov 02 22:21:45.027099 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.023971    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" event={"ID":"d2374b6f-76f5-42ed-a366-3173ffe13c3b","Type":"ContainerDied","Data":"47c155319b84e7189950cbf237692d80f35886fb70ba9ec47b027258f8171565"}
Nov 02 22:21:45.036382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.036272    4929 generic.go:356] "Generic (PLEG): container finished" podID="51d676b3-6802-4995-b9f0-44e42cbff2d3" containerID="32b74c28e341ffad413b260d84b9cd269dfa0a9901bc911d645625b3c30bece1" exitCode=0
Nov 02 22:21:45.036598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.036512    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" event={"ID":"51d676b3-6802-4995-b9f0-44e42cbff2d3","Type":"ContainerDied","Data":"32b74c28e341ffad413b260d84b9cd269dfa0a9901bc911d645625b3c30bece1"}
Nov 02 22:21:45.091945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.091917    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:21:45.094052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.094029    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:21:45.173036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.172995    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\" (UniqueName: \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\") pod \"51d676b3-6802-4995-b9f0-44e42cbff2d3\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") "
Nov 02 22:21:45.173243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.173065    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\" (UniqueName: \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\") pod \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") "
Nov 02 22:21:45.173243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.173092    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\") pod \"51d676b3-6802-4995-b9f0-44e42cbff2d3\" (UID: \"51d676b3-6802-4995-b9f0-44e42cbff2d3\") "
Nov 02 22:21:45.173243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.173208    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\") pod \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\" (UID: \"d2374b6f-76f5-42ed-a366-3173ffe13c3b\") "
Nov 02 22:21:45.173609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.173574    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume" pod "51d676b3-6802-4995-b9f0-44e42cbff2d3" (UID: "51d676b3-6802-4995-b9f0-44e42cbff2d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:21:45.173901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.173877    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume" pod "d2374b6f-76f5-42ed-a366-3173ffe13c3b" (UID: "d2374b6f-76f5-42ed-a366-3173ffe13c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:21:45.174267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.174239    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d676b3-6802-4995-b9f0-44e42cbff2d3-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:45.174367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.174268    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2374b6f-76f5-42ed-a366-3173ffe13c3b-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:45.175548 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.175508    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2" pod "d2374b6f-76f5-42ed-a366-3173ffe13c3b" (UID: "d2374b6f-76f5-42ed-a366-3173ffe13c3b"). InnerVolumeSpecName "kube-api-access-n8hm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:21:45.175653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.175552    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp" pod "51d676b3-6802-4995-b9f0-44e42cbff2d3" (UID: "51d676b3-6802-4995-b9f0-44e42cbff2d3"). InnerVolumeSpecName "kube-api-access-69rkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:21:45.275652 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.275573    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rkp\" (UniqueName: \"kubernetes.io/projected/51d676b3-6802-4995-b9f0-44e42cbff2d3-kube-api-access-69rkp\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:45.275652 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:45.275599    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hm2\" (UniqueName: \"kubernetes.io/projected/d2374b6f-76f5-42ed-a366-3173ffe13c3b-kube-api-access-n8hm2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:46.048056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.047966    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"
Nov 02 22:21:46.048056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.047998    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw" event={"ID":"51d676b3-6802-4995-b9f0-44e42cbff2d3","Type":"ContainerDied","Data":"1982431da863a69e2ebaf5ca294f8da3ffdef73e35b53ac6f4b9c5136632a5fc"}
Nov 02 22:21:46.048649 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.048067    4929 scope.go:122] "RemoveContainer" containerID="32b74c28e341ffad413b260d84b9cd269dfa0a9901bc911d645625b3c30bece1"
Nov 02 22:21:46.049895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.049873    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"
Nov 02 22:21:46.049999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.049914    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85" event={"ID":"d2374b6f-76f5-42ed-a366-3173ffe13c3b","Type":"ContainerDied","Data":"8977c23f738c992ddc87079908d474352985fb337d620ea46da29bfbb4f1864c"}
Nov 02 22:21:46.059796 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.059771    4929 scope.go:122] "RemoveContainer" containerID="47c155319b84e7189950cbf237692d80f35886fb70ba9ec47b027258f8171565"
Nov 02 22:21:46.074164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.074138    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:21:46.078221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.078197    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7144fe-5786cd4c55-6klnw"]
Nov 02 22:21:46.086673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.086644    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:21:46.090476 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.090454    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7144fe-f45574dff-c8n85"]
Nov 02 22:21:46.807112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.807073    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d676b3-6802-4995-b9f0-44e42cbff2d3" path="/var/lib/kubelet/pods/51d676b3-6802-4995-b9f0-44e42cbff2d3/volumes"
Nov 02 22:21:46.807666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:46.807647    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" path="/var/lib/kubelet/pods/d2374b6f-76f5-42ed-a366-3173ffe13c3b/volumes"
Nov 02 22:21:50.519593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:50.519557    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:21:50.520085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:50.519826    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" podUID="9088356e-6c6d-4d9b-99b5-91346071aee3" containerName="maas-api" containerID="cri-o://8475892560d0a15caabde785e04d9d403b455a03ff1f1e3ba3715fd2a73578dc" gracePeriod=30
Nov 02 22:21:51.126674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.126621    4929 generic.go:356] "Generic (PLEG): container finished" podID="9088356e-6c6d-4d9b-99b5-91346071aee3" containerID="8475892560d0a15caabde785e04d9d403b455a03ff1f1e3ba3715fd2a73578dc" exitCode=0
Nov 02 22:21:51.127206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.127180    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" event={"ID":"9088356e-6c6d-4d9b-99b5-91346071aee3","Type":"ContainerDied","Data":"8475892560d0a15caabde785e04d9d403b455a03ff1f1e3ba3715fd2a73578dc"}
Nov 02 22:21:51.127379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.127363    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b" event={"ID":"9088356e-6c6d-4d9b-99b5-91346071aee3","Type":"ContainerDied","Data":"4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4"}
Nov 02 22:21:51.127480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.127468    4929 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4090d631131540fdefdd9c6c44807a0dc219a556d1f581e85d8012ab90b640c4"
Nov 02 22:21:51.141934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.141906    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:21:51.246610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.246573    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\") pod \"9088356e-6c6d-4d9b-99b5-91346071aee3\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") "
Nov 02 22:21:51.246915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.246902    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\") pod \"9088356e-6c6d-4d9b-99b5-91346071aee3\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") "
Nov 02 22:21:51.246969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.246928    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\") pod \"9088356e-6c6d-4d9b-99b5-91346071aee3\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") "
Nov 02 22:21:51.246969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.246949    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\" (UniqueName: \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\") pod \"9088356e-6c6d-4d9b-99b5-91346071aee3\" (UID: \"9088356e-6c6d-4d9b-99b5-91346071aee3\") "
Nov 02 22:21:51.247367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.247328    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca" pod "9088356e-6c6d-4d9b-99b5-91346071aee3" (UID: "9088356e-6c6d-4d9b-99b5-91346071aee3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:21:51.247731 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.247705    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9088356e-6c6d-4d9b-99b5-91346071aee3-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:51.248869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.248833    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls" pod "9088356e-6c6d-4d9b-99b5-91346071aee3" (UID: "9088356e-6c6d-4d9b-99b5-91346071aee3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:21:51.249105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.249078    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs" pod "9088356e-6c6d-4d9b-99b5-91346071aee3" (UID: "9088356e-6c6d-4d9b-99b5-91346071aee3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:21:51.249369 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.249340    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7" pod "9088356e-6c6d-4d9b-99b5-91346071aee3" (UID: "9088356e-6c6d-4d9b-99b5-91346071aee3"). InnerVolumeSpecName "kube-api-access-8c9w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:21:51.348991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.348895    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:51.348991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.348927    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9w7\" (UniqueName: \"kubernetes.io/projected/9088356e-6c6d-4d9b-99b5-91346071aee3-kube-api-access-8c9w7\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:51.348991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:51.348939    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9088356e-6c6d-4d9b-99b5-91346071aee3-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:21:52.134460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:52.134423    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"
Nov 02 22:21:52.160580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:52.160543    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:21:52.165047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:52.165020    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7144fe-67d9ffd689-cdl9b"]
Nov 02 22:21:52.810611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:52.810576    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9088356e-6c6d-4d9b-99b5-91346071aee3" path="/var/lib/kubelet/pods/9088356e-6c6d-4d9b-99b5-91346071aee3/volumes"
Nov 02 22:21:56.585077 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.585032    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:21:56.586165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586139    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9088356e-6c6d-4d9b-99b5-91346071aee3" containerName="maas-api"
Nov 02 22:21:56.586165 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586163    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9088356e-6c6d-4d9b-99b5-91346071aee3" containerName="maas-api"
Nov 02 22:21:56.586358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586226    4929 cpu_manager.go:414] "containerMap: removing container" podUID="51d676b3-6802-4995-b9f0-44e42cbff2d3" containerName="payload-pre-processing"
Nov 02 22:21:56.586358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586234    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d676b3-6802-4995-b9f0-44e42cbff2d3" containerName="payload-pre-processing"
Nov 02 22:21:56.586358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586254    4929 cpu_manager.go:414] "containerMap: removing container" podUID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" containerName="payload-processing"
Nov 02 22:21:56.586358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586261    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" containerName="payload-processing"
Nov 02 22:21:56.586610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586494    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9088356e-6c6d-4d9b-99b5-91346071aee3" containerName="maas-api"
Nov 02 22:21:56.586610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586519    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d676b3-6802-4995-b9f0-44e42cbff2d3" containerName="payload-pre-processing"
Nov 02 22:21:56.586610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.586548    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2374b6f-76f5-42ed-a366-3173ffe13c3b" containerName="payload-processing"
Nov 02 22:21:56.592590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.592525    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.594127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.594095    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:21:56.597614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.597591    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5eba37\"/\"openshift-service-ca.crt\""
Nov 02 22:21:56.598115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.597953    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5eba37\"/\"test-model-5eba37-kserve-self-signed-certs\""
Nov 02 22:21:56.598247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.597953    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5eba37\"/\"kube-root-ca.crt\""
Nov 02 22:21:56.598354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.598023    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5eba37\"/\"default-dockercfg-6jcmm\""
Nov 02 22:21:56.645778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.645731    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.646081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.645924    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2bh\" (UniqueName: \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.646178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.646094    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.646178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.646147    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.646269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.646217    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.646418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.646388    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748400    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2bh\" (UniqueName: \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748564    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748616    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748665    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748790 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748782    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.748852 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.748844    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.749101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.749079    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.749183 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.749126    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.749308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.749283    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.751010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.750980    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.751578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.751556    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.755899 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.755876    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2bh\" (UniqueName: \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\") pod \"test-model-5eba37-kserve-55d476c4b9-wsl78\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") " pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:56.910918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:56.909667    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:21:57.219772 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:21:57.219732    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99917624_eb73_4e3a_a1f3_b504df6ca6e1.slice/crio-177904fcbd74c29034e00fcadd3334882428013920e8716033d7db376a504a68 WatchSource:0}: Error finding container 177904fcbd74c29034e00fcadd3334882428013920e8716033d7db376a504a68: Status 404 returned error can't find the container with id 177904fcbd74c29034e00fcadd3334882428013920e8716033d7db376a504a68
Nov 02 22:21:57.220899 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:57.220265    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:21:57.521455 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:21:57.521362    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:21:58.181947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.181904    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" event={"ID":"99917624-eb73-4e3a-a1f3-b504df6ca6e1","Type":"ContainerStarted","Data":"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"}
Nov 02 22:21:58.182452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.181957    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" event={"ID":"99917624-eb73-4e3a-a1f3-b504df6ca6e1","Type":"ContainerStarted","Data":"177904fcbd74c29034e00fcadd3334882428013920e8716033d7db376a504a68"}
Nov 02 22:21:58.201748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.201689    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" podStartSLOduration=2.201675064 podStartE2EDuration="2.201675064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:58.19744045 +0000 UTC m=+1455.505531256" watchObservedRunningTime="2026-11-02 22:21:58.201675064 +0000 UTC m=+1455.509765872"
Nov 02 22:21:58.891752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.891717    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:21:58.891962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.891938    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy" containerID="cri-o://c3aff3411aaa441820bc4fd3b9e0e7ead94e2668a322d2e023b34592effc2775" gracePeriod=30
Nov 02 22:21:58.910287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:21:58.910253    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:22:03.093218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:03.093172    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:22:04.386280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.386242    4929 generic.go:356] "Generic (PLEG): container finished" podID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerID="c3aff3411aaa441820bc4fd3b9e0e7ead94e2668a322d2e023b34592effc2775" exitCode=0
Nov 02 22:22:04.386797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.386766    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerDied","Data":"c3aff3411aaa441820bc4fd3b9e0e7ead94e2668a322d2e023b34592effc2775"}
Nov 02 22:22:04.386890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.386847    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7" event={"ID":"9b9a9a4c-f23d-4243-b857-fc5d55322114","Type":"ContainerDied","Data":"6bc3fbc436aa41ac784971066a1b1d5585038e3d6289f1cb50260bdf0b3952e8"}
Nov 02 22:22:04.386890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.386873    4929 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc3fbc436aa41ac784971066a1b1d5585038e3d6289f1cb50260bdf0b3952e8"
Nov 02 22:22:04.387007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.386891    4929 scope.go:122] "RemoveContainer" containerID="701d7efc37732cd8e4c19a3f4c199ffa945de7931a3fda379cc83c9212c68ebe"
Nov 02 22:22:04.390769 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.390740    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:22:04.452965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.452863    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.452965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.452926    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.452965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.452961    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453063    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453170    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453263 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453255    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453369    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453377    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:04.453529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453483    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453581    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\") pod \"9b9a9a4c-f23d-4243-b857-fc5d55322114\" (UID: \"9b9a9a4c-f23d-4243-b857-fc5d55322114\") "
Nov 02 22:22:04.453712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.453689    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:04.454196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454174    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:22:04.454439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454417    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:04.454653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454632    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.454736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454660    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.454736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454674    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9a9a4c-f23d-4243-b857-fc5d55322114-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.454736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454688    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.454918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.454770    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:04.457317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.457290    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "kube-api-access-z94rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:04.457463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.457354    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:04.458582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.457674    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:22:04.458582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.457976    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token" pod "9b9a9a4c-f23d-4243-b857-fc5d55322114" (UID: "9b9a9a4c-f23d-4243-b857-fc5d55322114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:04.556245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.556208    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.556245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.556240    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.556245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.556250    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.556492 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.556259    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/9b9a9a4c-f23d-4243-b857-fc5d55322114-kube-api-access-z94rg\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:04.556492 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:04.556268    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9a9a4c-f23d-4243-b857-fc5d55322114-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:05.424284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:05.424241    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"
Nov 02 22:22:05.497566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:05.485270    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:22:05.497566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:05.495716    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7144fe-openshift-default-7695b67fdf-wsmg7"]
Nov 02 22:22:06.807466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:06.807430    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" path="/var/lib/kubelet/pods/9b9a9a4c-f23d-4243-b857-fc5d55322114/volumes"
Nov 02 22:22:06.910598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:06.910554    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:22:06.923216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:06.923186    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:22:07.453128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:07.453100    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:22:09.487518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:09.487475    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:22:09.499366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:09.499330    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:22:14.487972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.487923    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" podUID="e99d0205-b9ce-41f2-857c-2610e6b62868" containerName="payload-processing" containerID="cri-o://16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce" gracePeriod=25
Nov 02 22:22:14.500675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.500640    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" podUID="5a28435b-cae8-4687-b858-d3f6796d44c7" containerName="payload-pre-processing" containerID="cri-o://e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37" gracePeriod=25
Nov 02 22:22:14.791565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.791523    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:22:14.855928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.855898    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:22:14.867582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.867523    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\" (UniqueName: \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\") pod \"e99d0205-b9ce-41f2-857c-2610e6b62868\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") "
Nov 02 22:22:14.867767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.867729    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\" (UniqueName: \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\") pod \"5a28435b-cae8-4687-b858-d3f6796d44c7\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") "
Nov 02 22:22:14.867857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.867773    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\") pod \"e99d0205-b9ce-41f2-857c-2610e6b62868\" (UID: \"e99d0205-b9ce-41f2-857c-2610e6b62868\") "
Nov 02 22:22:14.867961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.867944    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\") pod \"5a28435b-cae8-4687-b858-d3f6796d44c7\" (UID: \"5a28435b-cae8-4687-b858-d3f6796d44c7\") "
Nov 02 22:22:14.868853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.868355    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume" pod "5a28435b-cae8-4687-b858-d3f6796d44c7" (UID: "5a28435b-cae8-4687-b858-d3f6796d44c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:22:14.868853 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.868811    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a28435b-cae8-4687-b858-d3f6796d44c7-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:14.869457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.869425    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume" pod "e99d0205-b9ce-41f2-857c-2610e6b62868" (UID: "e99d0205-b9ce-41f2-857c-2610e6b62868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:22:14.870068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.870045    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55" pod "e99d0205-b9ce-41f2-857c-2610e6b62868" (UID: "e99d0205-b9ce-41f2-857c-2610e6b62868"). InnerVolumeSpecName "kube-api-access-fhq55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:14.870385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.870365    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6" pod "5a28435b-cae8-4687-b858-d3f6796d44c7" (UID: "5a28435b-cae8-4687-b858-d3f6796d44c7"). InnerVolumeSpecName "kube-api-access-wg9l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:14.970309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.970256    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9l6\" (UniqueName: \"kubernetes.io/projected/5a28435b-cae8-4687-b858-d3f6796d44c7-kube-api-access-wg9l6\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:14.970309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.970308    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d0205-b9ce-41f2-857c-2610e6b62868-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:14.970309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:14.970319    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhq55\" (UniqueName: \"kubernetes.io/projected/e99d0205-b9ce-41f2-857c-2610e6b62868-kube-api-access-fhq55\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:15.547616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.547581    4929 generic.go:356] "Generic (PLEG): container finished" podID="e99d0205-b9ce-41f2-857c-2610e6b62868" containerID="16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce" exitCode=0
Nov 02 22:22:15.548102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.547625    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"
Nov 02 22:22:15.548102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.547669    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" event={"ID":"e99d0205-b9ce-41f2-857c-2610e6b62868","Type":"ContainerDied","Data":"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"}
Nov 02 22:22:15.548102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.547734    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8" event={"ID":"e99d0205-b9ce-41f2-857c-2610e6b62868","Type":"ContainerDied","Data":"81aae65505f0b41eeb318d7723c15147acd8f20b521c0aebc0d05fdc3265aace"}
Nov 02 22:22:15.548102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.547769    4929 scope.go:122] "RemoveContainer" containerID="16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"
Nov 02 22:22:15.549360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.549329    4929 generic.go:356] "Generic (PLEG): container finished" podID="5a28435b-cae8-4687-b858-d3f6796d44c7" containerID="e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37" exitCode=0
Nov 02 22:22:15.549484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.549362    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" event={"ID":"5a28435b-cae8-4687-b858-d3f6796d44c7","Type":"ContainerDied","Data":"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"}
Nov 02 22:22:15.549484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.549391    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"
Nov 02 22:22:15.549484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.549428    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5" event={"ID":"5a28435b-cae8-4687-b858-d3f6796d44c7","Type":"ContainerDied","Data":"d37bf4b56d6a02276c41b05f3b321f1ecb1730c252cec44ec7cdcb6bc72a987c"}
Nov 02 22:22:15.560254 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.560236    4929 scope.go:122] "RemoveContainer" containerID="16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"
Nov 02 22:22:15.560592 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:15.560569    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce\": container with ID starting with 16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce not found: ID does not exist" containerID="16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"
Nov 02 22:22:15.560698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.560620    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce"} err="failed to get container status \"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce\": rpc error: code = NotFound desc = could not find container \"16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce\": container with ID starting with 16f212303b2b78e8d88159eedc07d36377c80c857406aab0b56d87301b9979ce not found: ID does not exist"
Nov 02 22:22:15.560698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.560642    4929 scope.go:122] "RemoveContainer" containerID="e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"
Nov 02 22:22:15.571754 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.571732    4929 scope.go:122] "RemoveContainer" containerID="e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"
Nov 02 22:22:15.572067 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:15.572047    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37\": container with ID starting with e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37 not found: ID does not exist" containerID="e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"
Nov 02 22:22:15.572155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.572075    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37"} err="failed to get container status \"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37\": rpc error: code = NotFound desc = could not find container \"e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37\": container with ID starting with e9c4c67c352aaeb18d5aa4dce755c66367b643328c925b6750e102676d5c6d37 not found: ID does not exist"
Nov 02 22:22:15.574935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.574911    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:22:15.579580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.579558    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-334860-6746c59647-2svs8"]
Nov 02 22:22:15.589777 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.589744    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:22:15.593466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:15.593446    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-334860-59b46dcc47-bchn5"]
Nov 02 22:22:16.806372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:16.806338    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a28435b-cae8-4687-b858-d3f6796d44c7" path="/var/lib/kubelet/pods/5a28435b-cae8-4687-b858-d3f6796d44c7/volumes"
Nov 02 22:22:16.806884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:16.806868    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99d0205-b9ce-41f2-857c-2610e6b62868" path="/var/lib/kubelet/pods/e99d0205-b9ce-41f2-857c-2610e6b62868/volumes"
Nov 02 22:22:18.970753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:18.970700    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:22:18.972599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:18.971063    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" podUID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" containerName="maas-api" containerID="cri-o://50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff" gracePeriod=30
Nov 02 22:22:19.417562 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.414758    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:22:19.513591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.512147    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\") pod \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") "
Nov 02 22:22:19.513591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.512242    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\") pod \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") "
Nov 02 22:22:19.513591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.512404    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\" (UniqueName: \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\") pod \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") "
Nov 02 22:22:19.513591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.512478    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\") pod \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\" (UID: \"757a2431-fba1-4c0f-ad05-e563ac8c8d47\") "
Nov 02 22:22:19.513940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.513773    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca" pod "757a2431-fba1-4c0f-ad05-e563ac8c8d47" (UID: "757a2431-fba1-4c0f-ad05-e563ac8c8d47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:22:19.536113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.528892    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757a2431-fba1-4c0f-ad05-e563ac8c8d47-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:19.536113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.531452    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs" pod "757a2431-fba1-4c0f-ad05-e563ac8c8d47" (UID: "757a2431-fba1-4c0f-ad05-e563ac8c8d47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:22:19.546222 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.544679    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b" pod "757a2431-fba1-4c0f-ad05-e563ac8c8d47" (UID: "757a2431-fba1-4c0f-ad05-e563ac8c8d47"). InnerVolumeSpecName "kube-api-access-m6r5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:19.551837 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.551761    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls" pod "757a2431-fba1-4c0f-ad05-e563ac8c8d47" (UID: "757a2431-fba1-4c0f-ad05-e563ac8c8d47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:22:19.584522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.584490    4929 generic.go:356] "Generic (PLEG): container finished" podID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" containerID="50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff" exitCode=0
Nov 02 22:22:19.584752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.584557    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"
Nov 02 22:22:19.584752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.584592    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" event={"ID":"757a2431-fba1-4c0f-ad05-e563ac8c8d47","Type":"ContainerDied","Data":"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"}
Nov 02 22:22:19.584752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.584658    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2" event={"ID":"757a2431-fba1-4c0f-ad05-e563ac8c8d47","Type":"ContainerDied","Data":"196a26cacd8a3fad0da27629fe9e703201ec0e4899378b55bfb7e98a55ea0de8"}
Nov 02 22:22:19.584752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.584687    4929 scope.go:122] "RemoveContainer" containerID="50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"
Nov 02 22:22:19.596605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.596512    4929 scope.go:122] "RemoveContainer" containerID="50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"
Nov 02 22:22:19.596909 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:19.596885    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff\": container with ID starting with 50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff not found: ID does not exist" containerID="50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"
Nov 02 22:22:19.596979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.596918    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff"} err="failed to get container status \"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff\": rpc error: code = NotFound desc = could not find container \"50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff\": container with ID starting with 50564b1aaf7e4bcdb2ef4cc1085180b65e34a50687ef47c92df24e3559b475ff not found: ID does not exist"
Nov 02 22:22:19.611395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.611368    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:22:19.615854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.615825    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-334860-66cd59cbf-kncg2"]
Nov 02 22:22:19.630297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.630262    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:19.630297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.630294    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6r5b\" (UniqueName: \"kubernetes.io/projected/757a2431-fba1-4c0f-ad05-e563ac8c8d47-kube-api-access-m6r5b\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:19.630480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:19.630311    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757a2431-fba1-4c0f-ad05-e563ac8c8d47-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:20.806810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:20.806777    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" path="/var/lib/kubelet/pods/757a2431-fba1-4c0f-ad05-e563ac8c8d47/volumes"
Nov 02 22:22:24.051983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.051948    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:22:24.052801 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052781    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.052855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052802    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.052855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052817    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5a28435b-cae8-4687-b858-d3f6796d44c7" containerName="payload-pre-processing"
Nov 02 22:22:24.052855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052823    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a28435b-cae8-4687-b858-d3f6796d44c7" containerName="payload-pre-processing"
Nov 02 22:22:24.052855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052850    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.052855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052855    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052870    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e99d0205-b9ce-41f2-857c-2610e6b62868" containerName="payload-processing"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052875    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99d0205-b9ce-41f2-857c-2610e6b62868" containerName="payload-processing"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052886    4929 cpu_manager.go:414] "containerMap: removing container" podUID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" containerName="maas-api"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052891    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" containerName="maas-api"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.052991    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="757a2431-fba1-4c0f-ad05-e563ac8c8d47" containerName="maas-api"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.053004    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.053011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.053013    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a28435b-cae8-4687-b858-d3f6796d44c7" containerName="payload-pre-processing"
Nov 02 22:22:24.053218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.053021    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9a9a4c-f23d-4243-b857-fc5d55322114" containerName="istio-proxy"
Nov 02 22:22:24.053218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.053027    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99d0205-b9ce-41f2-857c-2610e6b62868" containerName="payload-processing"
Nov 02 22:22:24.058029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.058009    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.064230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.064198    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a54217-openshift-default-dockercfg-7tr9f\""
Nov 02 22:22:24.066912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.066884    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:22:24.131567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131516    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131646    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131686    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131755    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131785    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131811    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvbl\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131849    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.131942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131892    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.132008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.131960    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.233616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233576    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.233810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233656    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.233810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233758    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.233810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233799    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233842    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvbl\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233892    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233959    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.233975    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234060    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234067    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234153    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234201    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234437    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.234756 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.234734    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.236188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.236160    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.236707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.236684    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.241281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.241252    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.241403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.241387    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvbl\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.394424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.394393    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:24.728189 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:22:24.728161    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7af1b66_40b9_4956_80f9_dc1f86cef997.slice/crio-936eb5e8bf72f9fb8e639b486609bc87d1cb960d1b3d5685407a9de11e92af25 WatchSource:0}: Error finding container 936eb5e8bf72f9fb8e639b486609bc87d1cb960d1b3d5685407a9de11e92af25: Status 404 returned error can't find the container with id 936eb5e8bf72f9fb8e639b486609bc87d1cb960d1b3d5685407a9de11e92af25
Nov 02 22:22:24.729277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.729246    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:22:24.731609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.731567    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:22:24.731703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.731653    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:22:24.731765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:24.731697    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:22:25.692244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:25.692201    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" event={"ID":"d7af1b66-40b9-4956-80f9-dc1f86cef997","Type":"ContainerStarted","Data":"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"}
Nov 02 22:22:25.693917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:25.693881    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" event={"ID":"d7af1b66-40b9-4956-80f9-dc1f86cef997","Type":"ContainerStarted","Data":"936eb5e8bf72f9fb8e639b486609bc87d1cb960d1b3d5685407a9de11e92af25"}
Nov 02 22:22:25.721967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:25.721912    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podStartSLOduration=1.721897528 podStartE2EDuration="1.721897528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:25.716908392 +0000 UTC m=+1483.024999196" watchObservedRunningTime="2026-11-02 22:22:25.721897528 +0000 UTC m=+1483.029988408"
Nov 02 22:22:26.395445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:26.395415    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:26.396959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:26.396934    4929 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 22:22:26.397101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:26.396989    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 22:22:27.227961 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.227926    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:22:27.228830 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.228759    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" containerID="cri-o://4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb" gracePeriod=30
Nov 02 22:22:27.427558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.417984    4929 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 22:22:27.427558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.418046    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 22:22:27.612364 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:27.612317    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:22:27.707437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.707405    4929 patch_prober.go:29] interesting pod/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:22:27.707680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:27.707463    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:22:28.394942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:28.394901    4929 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 22:22:28.395387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:28.394960    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 22:22:29.395737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:29.395699    4929 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 22:22:29.396242 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:29.395763    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 22:22:30.395240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:30.395199    4929 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 22:22:30.395419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:30.395257    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 22:22:31.542468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:31.542110    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:31.542960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:31.542702    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:31.544036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:31.544013    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:22:32.547973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.547946    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:22:32.622272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622230    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622288    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622319    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622635    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622674    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622707    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622804    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622859    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.622987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.622944    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\") pod \"a861bf8d-832d-4f34-8ae9-643e56c1f632\" (UID: \"a861bf8d-832d-4f34-8ae9-643e56c1f632\") "
Nov 02 22:22:32.623354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.623178    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:32.623817 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.623555    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:32.623997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.623974    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.624069 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.624005    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.624266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.624242    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:32.624683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.624658    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:22:32.625187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.625154    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:32.627512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.627485    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:32.627621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.627569    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:32.627786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.627762    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "kube-api-access-lsknw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:32.628983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.628947    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo" pod "a861bf8d-832d-4f34-8ae9-643e56c1f632" (UID: "a861bf8d-832d-4f34-8ae9-643e56c1f632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:22:32.704202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.704166    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:22:32.725757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725714    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a861bf8d-832d-4f34-8ae9-643e56c1f632-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725745    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725786    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725799    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsknw\" (UniqueName: \"kubernetes.io/projected/a861bf8d-832d-4f34-8ae9-643e56c1f632-kube-api-access-lsknw\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725816    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725830    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.725949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.725843    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a861bf8d-832d-4f34-8ae9-643e56c1f632-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:32.779198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.779104    4929 generic.go:356] "Generic (PLEG): container finished" podID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerID="4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb" exitCode=0
Nov 02 22:22:32.779198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.779160    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"
Nov 02 22:22:32.779198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.779188    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerDied","Data":"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"}
Nov 02 22:22:32.779526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.779270    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4" event={"ID":"a861bf8d-832d-4f34-8ae9-643e56c1f632","Type":"ContainerDied","Data":"ac9f84aa43f31cc3e344df78f8ec22b1955c419515bec8a652a147ee42a8f90e"}
Nov 02 22:22:32.779526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.779297    4929 scope.go:122] "RemoveContainer" containerID="4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"
Nov 02 22:22:32.792170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.792149    4929 scope.go:122] "RemoveContainer" containerID="d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"
Nov 02 22:22:32.812654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.812528    4929 scope.go:122] "RemoveContainer" containerID="4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"
Nov 02 22:22:32.813179 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:32.813142    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb\": container with ID starting with 4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb not found: ID does not exist" containerID="4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"
Nov 02 22:22:32.813298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.813191    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb"} err="failed to get container status \"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb\": rpc error: code = NotFound desc = could not find container \"4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb\": container with ID starting with 4f0631e2d9cdeb9f4a5f7e7187560b8f27e748b57d4ec631e01a3f7b8f2f13fb not found: ID does not exist"
Nov 02 22:22:32.813298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.813218    4929 scope.go:122] "RemoveContainer" containerID="d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"
Nov 02 22:22:32.813761 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:32.813647    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7\": container with ID starting with d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7 not found: ID does not exist" containerID="d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"
Nov 02 22:22:32.813761 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.813735    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7"} err="failed to get container status \"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7\": rpc error: code = NotFound desc = could not find container \"d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7\": container with ID starting with d7413494b3a861006f6c04d6d479591e4dbed4e64961d086d59a956035a0dba7 not found: ID does not exist"
Nov 02 22:22:32.819672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.819648    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:22:32.819802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:32.819685    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-334860-openshift-default-56844c759c-w4zd4"]
Nov 02 22:22:34.807512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:34.807474    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" path="/var/lib/kubelet/pods/a861bf8d-832d-4f34-8ae9-643e56c1f632/volumes"
Nov 02 22:22:38.173214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:38.171513    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:22:43.172656 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:43.172606    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing" containerID="cri-o://945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454" gracePeriod=25
Nov 02 22:22:45.543355 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:45.543320    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:22:45.547674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:45.547647    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:22:46.666609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:46.666575    4929 scope.go:122] "RemoveContainer" containerID="c3aff3411aaa441820bc4fd3b9e0e7ead94e2668a322d2e023b34592effc2775"
Nov 02 22:22:46.677709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:46.677687    4929 scope.go:122] "RemoveContainer" containerID="8475892560d0a15caabde785e04d9d403b455a03ff1f1e3ba3715fd2a73578dc"
Nov 02 22:22:47.726468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:47.726404    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main" containerID="cri-o://3576a6e2f6c957740ce8ff2acf52996b8e0d99b994ccb9e86cf3f52080467c4b" gracePeriod=45
Nov 02 22:22:47.918345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:47.918302    4929 generic.go:356] "Generic (PLEG): container finished" podID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerID="3576a6e2f6c957740ce8ff2acf52996b8e0d99b994ccb9e86cf3f52080467c4b" exitCode=0
Nov 02 22:22:47.918527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:47.918513    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" event={"ID":"284396bb-975e-4e74-bb8c-e928d90b67a9","Type":"ContainerDied","Data":"3576a6e2f6c957740ce8ff2acf52996b8e0d99b994ccb9e86cf3f52080467c4b"}
Nov 02 22:22:48.022957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.022929    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:22:48.176989 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.176942    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.177180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.177143    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.177310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.177285    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.177578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.177391    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\" (UniqueName: \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.177578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.177506    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.177785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.177634    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\") pod \"284396bb-975e-4e74-bb8c-e928d90b67a9\" (UID: \"284396bb-975e-4e74-bb8c-e928d90b67a9\") "
Nov 02 22:22:48.178884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.178849    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:48.179348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.178882    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:48.179348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.179077    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:48.182003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.181929    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:22:48.182267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.182240    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:22:48.182749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.182724    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw" pod "284396bb-975e-4e74-bb8c-e928d90b67a9" (UID: "284396bb-975e-4e74-bb8c-e928d90b67a9"). InnerVolumeSpecName "kube-api-access-wgzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304148    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgzgw\" (UniqueName: \"kubernetes.io/projected/284396bb-975e-4e74-bb8c-e928d90b67a9-kube-api-access-wgzgw\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304195    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304211    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304225    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304237    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284396bb-975e-4e74-bb8c-e928d90b67a9-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.304258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.304251    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284396bb-975e-4e74-bb8c-e928d90b67a9-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:22:48.930411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.930368    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb" event={"ID":"284396bb-975e-4e74-bb8c-e928d90b67a9","Type":"ContainerDied","Data":"7a02e1dcd3dcf7366412a75cd5e3a153fa6d0e3ca87493b3906f259362f25fbf"}
Nov 02 22:22:48.931121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.930983    4929 scope.go:122] "RemoveContainer" containerID="3576a6e2f6c957740ce8ff2acf52996b8e0d99b994ccb9e86cf3f52080467c4b"
Nov 02 22:22:48.931121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.930394    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"
Nov 02 22:22:48.952607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.952573    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:22:48.956725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:48.956693    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/simulator-w4-c302b6-kserve-58c59585cf-jsrdb"]
Nov 02 22:22:50.608033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:50.607992    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:22:50.807565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:22:50.807513    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" path="/var/lib/kubelet/pods/284396bb-975e-4e74-bb8c-e928d90b67a9/volumes"
Nov 02 22:22:57.498064 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:22:57.498024    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:23:00.607791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:00.607752    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:23:01.579759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:01.579726    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:23:03.480002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.479974    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:23:03.588688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.588648    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\" (UniqueName: \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\") pod \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") "
Nov 02 22:23:03.588865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.588842    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\") pod \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\" (UID: \"5a3637ea-9eed-432f-a7b5-0c737b3d9055\") "
Nov 02 22:23:03.589294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.589266    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume" pod "5a3637ea-9eed-432f-a7b5-0c737b3d9055" (UID: "5a3637ea-9eed-432f-a7b5-0c737b3d9055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:23:03.589412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.589395    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3637ea-9eed-432f-a7b5-0c737b3d9055-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:03.590781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.590757    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44" pod "5a3637ea-9eed-432f-a7b5-0c737b3d9055" (UID: "5a3637ea-9eed-432f-a7b5-0c737b3d9055"). InnerVolumeSpecName "kube-api-access-8vw44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:03.691785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:03.691749    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vw44\" (UniqueName: \"kubernetes.io/projected/5a3637ea-9eed-432f-a7b5-0c737b3d9055-kube-api-access-8vw44\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:04.060088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.060047    4929 generic.go:356] "Generic (PLEG): container finished" podID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerID="945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454" exitCode=0
Nov 02 22:23:04.060256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.060104    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"
Nov 02 22:23:04.060256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.060133    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" event={"ID":"5a3637ea-9eed-432f-a7b5-0c737b3d9055","Type":"ContainerDied","Data":"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"}
Nov 02 22:23:04.060256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.060188    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr" event={"ID":"5a3637ea-9eed-432f-a7b5-0c737b3d9055","Type":"ContainerDied","Data":"b4b85cce71f2cec5f07f586e9c24e55aae767b66ae6449aa8ec8be9c193e9140"}
Nov 02 22:23:04.060256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.060210    4929 scope.go:122] "RemoveContainer" containerID="945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"
Nov 02 22:23:04.071319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.071300    4929 scope.go:122] "RemoveContainer" containerID="945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"
Nov 02 22:23:04.071614 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:04.071594    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454\": container with ID starting with 945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454 not found: ID does not exist" containerID="945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"
Nov 02 22:23:04.071683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.071624    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454"} err="failed to get container status \"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454\": rpc error: code = NotFound desc = could not find container \"945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454\": container with ID starting with 945c4f20a11d43777cf1379c65276d09da21fb4e5c13d0c231575359f66d2454 not found: ID does not exist"
Nov 02 22:23:04.084665 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.084637    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:23:04.091834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.091811    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c302b6-674cb85bb7-gh7dr"]
Nov 02 22:23:04.807380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:04.807344    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" path="/var/lib/kubelet/pods/5a3637ea-9eed-432f-a7b5-0c737b3d9055/volumes"
Nov 02 22:23:15.467039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.466994    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:23:15.468258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.467354    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy" containerID="cri-o://69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f" gracePeriod=30
Nov 02 22:23:15.475496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.475471    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:23:15.477022 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.476991    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy" containerID="cri-o://61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911" gracePeriod=30
Nov 02 22:23:15.481669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.481635    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:23:15.481885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:15.481861    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy" containerID="cri-o://d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15" gracePeriod=30
Nov 02 22:23:20.816331 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.816295    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:23:20.866323 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.866288    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:23:20.871185 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.871160    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:23:20.952295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952261    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.952470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952338    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.952470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952414    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.952470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952451    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.952715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952480    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.952715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952589    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.952715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952623    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.952715 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952676    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.952922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952740    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.952922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952775    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.952922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952847    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.952922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952887    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.953130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.952973    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953007    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953049    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953082    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953229    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.953330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953246    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.953330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953288    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.953493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953335    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.953493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953422    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.953493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953464    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.953683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953506    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.953683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953563    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\") pod \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\" (UID: \"ceb72af2-8e22-42c1-8633-b9ef19ca0b01\") "
Nov 02 22:23:20.953683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953564    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:23:20.953683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953619    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953706    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.953889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953774    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\") pod \"bbc6f219-bc05-4252-87d1-535f3dffee4c\" (UID: \"bbc6f219-bc05-4252-87d1-535f3dffee4c\") "
Nov 02 22:23:20.953889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.953827    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\") pod \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\" (UID: \"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9\") "
Nov 02 22:23:20.954113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.954090    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.954683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.954656    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.956254    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.956279    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc6f219-bc05-4252-87d1-535f3dffee4c-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.956293    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.956307    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.956992    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.957711    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.957903    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.957944    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.959618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.959593    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.959794    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.960203    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "kube-api-access-d6wdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.961578    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.961661    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.961789    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.961884    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "kube-api-access-vrg2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.962164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.962029    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.962173    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.962448    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.962637    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.962863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.962837    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.963035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.963014    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:23:20.963350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.963317    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:23:20.963692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.963674    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token" pod "ceb72af2-8e22-42c1-8633-b9ef19ca0b01" (UID: "ceb72af2-8e22-42c1-8633-b9ef19ca0b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.963957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.963929    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "kube-api-access-8mmx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:23:20.964294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.964270    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:23:20.964294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.964284    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo" pod "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" (UID: "bedfa64a-113a-4cdc-bcb6-5e84b818e8f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:23:20.964416 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:20.964350    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo" pod "bbc6f219-bc05-4252-87d1-535f3dffee4c" (UID: "bbc6f219-bc05-4252-87d1-535f3dffee4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057381    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057413    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057424    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057432    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057441    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057449    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057458    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057467    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057476    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057484    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057502    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057510    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057518    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057529    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrg2w\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-kube-api-access-vrg2w\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057581    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057592    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmx9\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-kube-api-access-8mmx9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057600    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057609    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057620    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057634    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc6f219-bc05-4252-87d1-535f3dffee4c-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057651    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wdn\" (UniqueName: \"kubernetes.io/projected/ceb72af2-8e22-42c1-8633-b9ef19ca0b01-kube-api-access-d6wdn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057661    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.057894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.057670    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc6f219-bc05-4252-87d1-535f3dffee4c-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:23:21.255480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.255425    4929 generic.go:356] "Generic (PLEG): container finished" podID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerID="d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15" exitCode=0
Nov 02 22:23:21.255739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.255609    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" event={"ID":"ceb72af2-8e22-42c1-8633-b9ef19ca0b01","Type":"ContainerDied","Data":"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"}
Nov 02 22:23:21.255739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.255705    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd" event={"ID":"ceb72af2-8e22-42c1-8633-b9ef19ca0b01","Type":"ContainerDied","Data":"f3e4963048f460ab32a1688bb649491408d931c7b46a47040d128c3e1c73ed40"}
Nov 02 22:23:21.255739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.255735    4929 scope.go:122] "RemoveContainer" containerID="d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"
Nov 02 22:23:21.256217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.256152    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"
Nov 02 22:23:21.258328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.257891    4929 generic.go:356] "Generic (PLEG): container finished" podID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerID="61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911" exitCode=0
Nov 02 22:23:21.258328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.258101    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"
Nov 02 22:23:21.258328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.258188    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" event={"ID":"bbc6f219-bc05-4252-87d1-535f3dffee4c","Type":"ContainerDied","Data":"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"}
Nov 02 22:23:21.258328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.258242    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg" event={"ID":"bbc6f219-bc05-4252-87d1-535f3dffee4c","Type":"ContainerDied","Data":"fcb2601840c673a1b73ed254583a0bc0081139ec57d2aaa2d3ad85ded318249e"}
Nov 02 22:23:21.266090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.266052    4929 generic.go:356] "Generic (PLEG): container finished" podID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerID="69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f" exitCode=0
Nov 02 22:23:21.266232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.266137    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" event={"ID":"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9","Type":"ContainerDied","Data":"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"}
Nov 02 22:23:21.266232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.266176    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk" event={"ID":"bedfa64a-113a-4cdc-bcb6-5e84b818e8f9","Type":"ContainerDied","Data":"e4e8353a262eb21500a0e5fe757c74eee4d334d7987de58005056891131ae72a"}
Nov 02 22:23:21.266408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.266385    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"
Nov 02 22:23:21.286764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.286449    4929 scope.go:122] "RemoveContainer" containerID="d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"
Nov 02 22:23:21.286949 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:21.286917    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15\": container with ID starting with d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15 not found: ID does not exist" containerID="d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"
Nov 02 22:23:21.287020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.286962    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15"} err="failed to get container status \"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15\": rpc error: code = NotFound desc = could not find container \"d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15\": container with ID starting with d03efbb6c736883bed34f70995ad63cd96f4644ac12f4f7c45d27760b21b2f15 not found: ID does not exist"
Nov 02 22:23:21.287020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.286989    4929 scope.go:122] "RemoveContainer" containerID="61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"
Nov 02 22:23:21.312435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.312366    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:23:21.316127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.316102    4929 scope.go:122] "RemoveContainer" containerID="61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"
Nov 02 22:23:21.316590 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:21.316517    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911\": container with ID starting with 61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911 not found: ID does not exist" containerID="61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"
Nov 02 22:23:21.316706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.316586    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911"} err="failed to get container status \"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911\": rpc error: code = NotFound desc = could not find container \"61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911\": container with ID starting with 61abb2df8d00192ece9f6eaba7d78d4c710c7cfeeb6ff8cd710e8b02c4096911 not found: ID does not exist"
Nov 02 22:23:21.316706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.316609    4929 scope.go:122] "RemoveContainer" containerID="69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"
Nov 02 22:23:21.317519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.317500    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-cdzfd"]
Nov 02 22:23:21.322605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.322581    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:23:21.326966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.326941    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-2wkrk"]
Nov 02 22:23:21.329010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.328988    4929 scope.go:122] "RemoveContainer" containerID="69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"
Nov 02 22:23:21.329371 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:21.329347    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f\": container with ID starting with 69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f not found: ID does not exist" containerID="69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"
Nov 02 22:23:21.329458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.329381    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f"} err="failed to get container status \"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f\": rpc error: code = NotFound desc = could not find container \"69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f\": container with ID starting with 69b15a4f0858d6b15b1b432a9a0e9cd25b85ea37caa95191da73601f5c4e1e8f not found: ID does not exist"
Nov 02 22:23:21.341024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.340996    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:23:21.347901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:21.347874    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-qkxtg"]
Nov 02 22:23:22.806729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:22.806694    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" path="/var/lib/kubelet/pods/bbc6f219-bc05-4252-87d1-535f3dffee4c/volumes"
Nov 02 22:23:22.807266 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:22.807251    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" path="/var/lib/kubelet/pods/bedfa64a-113a-4cdc-bcb6-5e84b818e8f9/volumes"
Nov 02 22:23:22.807749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:22.807735    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" path="/var/lib/kubelet/pods/ceb72af2-8e22-42c1-8633-b9ef19ca0b01/volumes"
Nov 02 22:23:27.497780 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:27.497740    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:23:46.786874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:46.786838    4929 scope.go:122] "RemoveContainer" containerID="073aa473221ef348f094f7aaff740585ff315b09773bc06eea8d1b8398f6beae"
Nov 02 22:23:51.319239 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:23:51.319200    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:23:57.501912 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:23:57.501819    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:24:21.599631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:21.599596    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:24:27.498598 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:24:27.498559    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:24:37.701984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:37.701942    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:24:48.121079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:48.121044    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:24:48.163918 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:48.157003    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:24:51.339751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:51.339707    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:24:52.725667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:52.725619    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerName="main" containerID="cri-o://53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397" gracePeriod=45
Nov 02 22:24:52.759915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:52.759883    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": dial tcp 10.133.0.151:8000: connect: connection refused"
Nov 02 22:24:53.057073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.057038    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:24:53.162840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.162795    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing" containerID="cri-o://47573edb14c6dbb56412cc050cc9348b6ec0f42b4ba77d7a83c01bb07329ad0b" gracePeriod=25
Nov 02 22:24:53.165343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.165289    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing" containerID="cri-o://1a38fd7a14779c1c6ec53ba9edfdfbe6ccc770def2ffa7d1f36bab54ce733978" gracePeriod=25
Nov 02 22:24:53.202522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.202485    4929 generic.go:356] "Generic (PLEG): container finished" podID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerID="53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397" exitCode=0
Nov 02 22:24:53.202758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.202566    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"
Nov 02 22:24:53.202758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.202561    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" event={"ID":"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda","Type":"ContainerDied","Data":"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"}
Nov 02 22:24:53.202758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.202671    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn" event={"ID":"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda","Type":"ContainerDied","Data":"47ad2bef8da7d9d42381d4d51917c9163a2a211c02f780948ac6d78a1d09ae94"}
Nov 02 22:24:53.202758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.202690    4929 scope.go:122] "RemoveContainer" containerID="53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"
Nov 02 22:24:53.203753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203667    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.203753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203742    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.203892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203780    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.203892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203839    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\" (UniqueName: \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.203998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203960    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:24:53.204055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.203986    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.204055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.204023    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:24:53.204199 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.204173    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:24:53.204319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.204232    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\") pod \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\" (UID: \"1ae9d89a-7ecc-4671-a704-31d4a9dc0cda\") "
Nov 02 22:24:53.205198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.205173    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.205316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.205228    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.205316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.205244    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.206236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.206204    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:24:53.206360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.206276    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "kube-api-access-twgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:24:53.206426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.206378    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs" pod "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" (UID: "1ae9d89a-7ecc-4671-a704-31d4a9dc0cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:24:53.238200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.238177    4929 scope.go:122] "RemoveContainer" containerID="53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"
Nov 02 22:24:53.238600 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:24:53.238575    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397\": container with ID starting with 53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397 not found: ID does not exist" containerID="53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"
Nov 02 22:24:53.238696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.238612    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397"} err="failed to get container status \"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397\": rpc error: code = NotFound desc = could not find container \"53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397\": container with ID starting with 53d7cc9158591c7bdd292a2256b42b641a05bc8475e7c369ea5d072a68d3f397 not found: ID does not exist"
Nov 02 22:24:53.307405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.307321    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.307405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.307356    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgwm\" (UniqueName: \"kubernetes.io/projected/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-kube-api-access-twgwm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.307405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.307368    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:53.527495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.527461    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:24:53.531483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:53.531459    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e49edd/test-model-e49edd-kserve-85c648dbb5-x7bjn"]
Nov 02 22:24:54.807940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:54.807906    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" path="/var/lib/kubelet/pods/1ae9d89a-7ecc-4671-a704-31d4a9dc0cda/volumes"
Nov 02 22:24:57.497368 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:24:57.497332    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:24:57.546327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:57.546288    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:24:58.536979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:58.536941    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:24:58.555395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:24:58.555346    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:05.326396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:05.326353    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:25:06.367616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.367568    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main" containerID="cri-o://07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1" gracePeriod=45
Nov 02 22:25:06.678247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.678216    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:25:06.711085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711025    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711275 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711106    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711344    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711451    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711487    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\" (UniqueName: \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711523    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\") pod \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\" (UID: \"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd\") "
Nov 02 22:25:06.711717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711689    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:06.711990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711965    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:06.711990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.711976    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:06.712235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.712212    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:06.712352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.712248    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:06.712352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.712259    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:06.714057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.714032    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:06.714179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.714127    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:06.714284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.714255    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd" pod "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" (UID: "a0e46b0c-36d0-4a2e-9183-3d904bacf3bd"). InnerVolumeSpecName "kube-api-access-lx7vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:06.813635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.813605    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7vd\" (UniqueName: \"kubernetes.io/projected/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-kube-api-access-lx7vd\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:06.813635 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.813634    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:06.813836 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:06.813648    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:07.355283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.355250    4929 generic.go:356] "Generic (PLEG): container finished" podID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerID="07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1" exitCode=0
Nov 02 22:25:07.355477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.355302    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"
Nov 02 22:25:07.355477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.355334    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" event={"ID":"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd","Type":"ContainerDied","Data":"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"}
Nov 02 22:25:07.355477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.355383    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz" event={"ID":"a0e46b0c-36d0-4a2e-9183-3d904bacf3bd","Type":"ContainerDied","Data":"a9d9a4fafe79faaf66e03a36389771043f82826f4d236dd2da4f29b414ea714e"}
Nov 02 22:25:07.355477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.355405    4929 scope.go:122] "RemoveContainer" containerID="07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"
Nov 02 22:25:07.365795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.365777    4929 scope.go:122] "RemoveContainer" containerID="07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"
Nov 02 22:25:07.366074 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:07.366054    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1\": container with ID starting with 07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1 not found: ID does not exist" containerID="07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"
Nov 02 22:25:07.366121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.366084    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1"} err="failed to get container status \"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1\": rpc error: code = NotFound desc = could not find container \"07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1\": container with ID starting with 07e8871de08bb4dc20643ab24ac688fb800c95b4672ff3c82eadd4799bec10e1 not found: ID does not exist"
Nov 02 22:25:07.374105 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.374082    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:25:07.377665 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.377642    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e30968/test-model-e30968-kserve-5bf44668b9-8gjnz"]
Nov 02 22:25:07.546644 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:07.546606    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:08.555743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.555698    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:08.727812 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.727779    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:25:08.792175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.792145    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:25:08.792395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.792372    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy" containerID="cri-o://a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d" gracePeriod=30
Nov 02 22:25:08.811748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.811672    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" path="/var/lib/kubelet/pods/a0e46b0c-36d0-4a2e-9183-3d904bacf3bd/volumes"
Nov 02 22:25:08.812159 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.812121    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:25:08.812477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:08.812452    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy" containerID="cri-o://f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7" gracePeriod=30
Nov 02 22:25:12.739502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:12.739457    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:25:13.055763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.055677    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:25:13.055971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.055937    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" podUID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" containerName="maas-api" containerID="cri-o://47c44f393eee673498bcbdf6e2e5784f31bf27579e0fbc1eeb342b88c6ddc3e2" gracePeriod=30
Nov 02 22:25:13.482353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.482315    4929 generic.go:356] "Generic (PLEG): container finished" podID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerID="1a38fd7a14779c1c6ec53ba9edfdfbe6ccc770def2ffa7d1f36bab54ce733978" exitCode=0
Nov 02 22:25:13.482882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.482853    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" event={"ID":"6a017c56-f362-43c3-9efd-3461e8dcb554","Type":"ContainerDied","Data":"1a38fd7a14779c1c6ec53ba9edfdfbe6ccc770def2ffa7d1f36bab54ce733978"}
Nov 02 22:25:13.485297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.485270    4929 generic.go:356] "Generic (PLEG): container finished" podID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" containerID="47c44f393eee673498bcbdf6e2e5784f31bf27579e0fbc1eeb342b88c6ddc3e2" exitCode=0
Nov 02 22:25:13.485802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.485556    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" event={"ID":"354a47a7-8241-4d2a-a09c-dc75aaa12cd5","Type":"ContainerDied","Data":"47c44f393eee673498bcbdf6e2e5784f31bf27579e0fbc1eeb342b88c6ddc3e2"}
Nov 02 22:25:13.490192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.490170    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" event={"ID":"123aac54-2085-49f4-8e27-483ccf274a1e","Type":"ContainerDied","Data":"47573edb14c6dbb56412cc050cc9348b6ec0f42b4ba77d7a83c01bb07329ad0b"}
Nov 02 22:25:13.490359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.488684    4929 generic.go:356] "Generic (PLEG): container finished" podID="123aac54-2085-49f4-8e27-483ccf274a1e" containerID="47573edb14c6dbb56412cc050cc9348b6ec0f42b4ba77d7a83c01bb07329ad0b" exitCode=0
Nov 02 22:25:13.563732 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.563323    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" containerName="main" containerID="cri-o://b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0" gracePeriod=45
Nov 02 22:25:13.677907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.677875    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": dial tcp 10.133.0.154:8000: connect: connection refused"
Nov 02 22:25:13.725395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.725349    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:25:13.796005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.795248    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\" (UniqueName: \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\") pod \"6a017c56-f362-43c3-9efd-3461e8dcb554\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") "
Nov 02 22:25:13.796005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.795317    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\") pod \"6a017c56-f362-43c3-9efd-3461e8dcb554\" (UID: \"6a017c56-f362-43c3-9efd-3461e8dcb554\") "
Nov 02 22:25:13.796960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.796702    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume" pod "6a017c56-f362-43c3-9efd-3461e8dcb554" (UID: "6a017c56-f362-43c3-9efd-3461e8dcb554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:13.804630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.804319    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b" pod "6a017c56-f362-43c3-9efd-3461e8dcb554" (UID: "6a017c56-f362-43c3-9efd-3461e8dcb554"). InnerVolumeSpecName "kube-api-access-djn5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:13.898983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.898947    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn5b\" (UniqueName: \"kubernetes.io/projected/6a017c56-f362-43c3-9efd-3461e8dcb554-kube-api-access-djn5b\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:13.899134 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.899004    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a017c56-f362-43c3-9efd-3461e8dcb554-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:13.941493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.939813    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:25:13.942326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.942246    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:25:13.976371 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:13.975889    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:25:14.102346 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.102311    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\") pod \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") "
Nov 02 22:25:14.102776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.102755    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.102917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.102796    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\" (UniqueName: \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.102978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.102922    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.103055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103038    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") pod \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") "
Nov 02 22:25:14.103115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103076    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\") pod \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") "
Nov 02 22:25:14.103176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103109    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\" (UniqueName: \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.103176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103164    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\") pod \"123aac54-2085-49f4-8e27-483ccf274a1e\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") "
Nov 02 22:25:14.103279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103206    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.103279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103269    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\" (UniqueName: \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\") pod \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\" (UID: \"354a47a7-8241-4d2a-a09c-dc75aaa12cd5\") "
Nov 02 22:25:14.103382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103299    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\") pod \"48e8a816-11c1-4d18-8487-87097f71cb93\" (UID: \"48e8a816-11c1-4d18-8487-87097f71cb93\") "
Nov 02 22:25:14.103452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103433    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\" (UniqueName: \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\") pod \"123aac54-2085-49f4-8e27-483ccf274a1e\" (UID: \"123aac54-2085-49f4-8e27-483ccf274a1e\") "
Nov 02 22:25:14.105505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103071    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.105505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.103289    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.105505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.104804    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca" pod "354a47a7-8241-4d2a-a09c-dc75aaa12cd5" (UID: "354a47a7-8241-4d2a-a09c-dc75aaa12cd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:14.105505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.105203    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume" pod "123aac54-2085-49f4-8e27-483ccf274a1e" (UID: "123aac54-2085-49f4-8e27-483ccf274a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:14.105505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.105467    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.106333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.106310    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.106420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.106339    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.106420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.106354    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/123aac54-2085-49f4-8e27-483ccf274a1e-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.106420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.106367    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.106420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.106380    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.109304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.109256    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls" pod "354a47a7-8241-4d2a-a09c-dc75aaa12cd5" (UID: "354a47a7-8241-4d2a-a09c-dc75aaa12cd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:14.112194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.112170    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s" pod "354a47a7-8241-4d2a-a09c-dc75aaa12cd5" (UID: "354a47a7-8241-4d2a-a09c-dc75aaa12cd5"). InnerVolumeSpecName "kube-api-access-4bw5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.112292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.112210    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "kube-api-access-nkhx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.112832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.112793    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:14.113096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.113033    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs" pod "354a47a7-8241-4d2a-a09c-dc75aaa12cd5" (UID: "354a47a7-8241-4d2a-a09c-dc75aaa12cd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:14.114131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.114093    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h" pod "123aac54-2085-49f4-8e27-483ccf274a1e" (UID: "123aac54-2085-49f4-8e27-483ccf274a1e"). InnerVolumeSpecName "kube-api-access-gvx8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.114433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.114414    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm" pod "48e8a816-11c1-4d18-8487-87097f71cb93" (UID: "48e8a816-11c1-4d18-8487-87097f71cb93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.136854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.136822    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:25:14.206907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.206872    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.206977    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207015    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207076    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207102    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207304 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207270    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207304    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:14.207434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207416    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207484    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207547    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\") pod \"9f8970e7-a632-48d5-9f65-d7adf860c52c\" (UID: \"9f8970e7-a632-48d5-9f65-d7adf860c52c\") "
Nov 02 22:25:14.207855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.207831    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.208024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208006    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208127    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208150    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e8a816-11c1-4d18-8487-87097f71cb93-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208163    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8970e7-a632-48d5-9f65-d7adf860c52c-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208177    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208187    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208196    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhx9\" (UniqueName: \"kubernetes.io/projected/48e8a816-11c1-4d18-8487-87097f71cb93-kube-api-access-nkhx9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208205    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e8a816-11c1-4d18-8487-87097f71cb93-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208213    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bw5s\" (UniqueName: \"kubernetes.io/projected/354a47a7-8241-4d2a-a09c-dc75aaa12cd5-kube-api-access-4bw5s\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208223    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208237    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvx8h\" (UniqueName: \"kubernetes.io/projected/123aac54-2085-49f4-8e27-483ccf274a1e-kube-api-access-gvx8h\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.208520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.208163    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.209661 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.209632    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.210730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.210704    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:14.210883 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.210796    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "kube-api-access-fwsxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.210883 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.210833    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.211237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.211215    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy" pod "9f8970e7-a632-48d5-9f65-d7adf860c52c" (UID: "9f8970e7-a632-48d5-9f65-d7adf860c52c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.251696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.251670    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:25:14.309824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.309790    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.309868    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.309903    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.309940    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310011    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310080    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310107    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.310177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310117    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310175    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310216    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\") pod \"89759db7-7ede-4cd4-9cc0-7b08206cb781\" (UID: \"89759db7-7ede-4cd4-9cc0-7b08206cb781\") "
Nov 02 22:25:14.310460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310274    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.310460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310316    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.310460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310351    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:14.310732 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.310509    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.311122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311096    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311128    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwsxb\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-kube-api-access-fwsxb\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311144    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311155    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311168    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311182    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89759db7-7ede-4cd4-9cc0-7b08206cb781-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311192    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311201    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311209    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311217    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.311233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.311226    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8970e7-a632-48d5-9f65-d7adf860c52c-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.313046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.313022    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.313046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.313029    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:14.313226 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.313072    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "kube-api-access-4b7qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:14.313662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.313637    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy" pod "89759db7-7ede-4cd4-9cc0-7b08206cb781" (UID: "89759db7-7ede-4cd4-9cc0-7b08206cb781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:14.413012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.412921    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.413012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.412956    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.413012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.412969    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b7qz\" (UniqueName: \"kubernetes.io/projected/89759db7-7ede-4cd4-9cc0-7b08206cb781-kube-api-access-4b7qz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.413012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.412979    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89759db7-7ede-4cd4-9cc0-7b08206cb781-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:14.499856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.499822    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"
Nov 02 22:25:14.500018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.499828    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct" event={"ID":"6a017c56-f362-43c3-9efd-3461e8dcb554","Type":"ContainerDied","Data":"4dcded819bc6de1a062f5c2ff81afadf34f0d0c5fea185f38b4a699b9cbddec9"}
Nov 02 22:25:14.500018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.500009    4929 scope.go:122] "RemoveContainer" containerID="1a38fd7a14779c1c6ec53ba9edfdfbe6ccc770def2ffa7d1f36bab54ce733978"
Nov 02 22:25:14.501354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.501332    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"
Nov 02 22:25:14.501354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.501338    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h" event={"ID":"354a47a7-8241-4d2a-a09c-dc75aaa12cd5","Type":"ContainerDied","Data":"08e4f485d2cac4d8de659ccef311783a761be1599c8700cd97be61108d9bdbe8"}
Nov 02 22:25:14.502919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.502893    4929 generic.go:356] "Generic (PLEG): container finished" podID="48e8a816-11c1-4d18-8487-87097f71cb93" containerID="b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0" exitCode=0
Nov 02 22:25:14.503032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.502929    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" event={"ID":"48e8a816-11c1-4d18-8487-87097f71cb93","Type":"ContainerDied","Data":"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"}
Nov 02 22:25:14.503032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.502973    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn" event={"ID":"48e8a816-11c1-4d18-8487-87097f71cb93","Type":"ContainerDied","Data":"626640f411050cf9456e4f1637fefcdcbb5b8a180a5f375361897db8ef8d76e2"}
Nov 02 22:25:14.503245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.503227    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"
Nov 02 22:25:14.504585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.504563    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78" event={"ID":"123aac54-2085-49f4-8e27-483ccf274a1e","Type":"ContainerDied","Data":"43b1d673e1d8e374805f18331f7f41e792937e968ed5ab592e9bcbf4922365cc"}
Nov 02 22:25:14.504689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.504595    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"
Nov 02 22:25:14.506763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.506741    4929 generic.go:356] "Generic (PLEG): container finished" podID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerID="f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7" exitCode=0
Nov 02 22:25:14.506873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.506844    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerDied","Data":"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"}
Nov 02 22:25:14.506948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.506883    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx" event={"ID":"89759db7-7ede-4cd4-9cc0-7b08206cb781","Type":"ContainerDied","Data":"8f2c032897dbee0350ff350d4eff113bcbfaa47a8d8943f1ade80e0abcf54bfe"}
Nov 02 22:25:14.507011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.506955    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"
Nov 02 22:25:14.512999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.512887    4929 generic.go:356] "Generic (PLEG): container finished" podID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerID="a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d" exitCode=0
Nov 02 22:25:14.512999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.512979    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"
Nov 02 22:25:14.513249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.512940    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" event={"ID":"9f8970e7-a632-48d5-9f65-d7adf860c52c","Type":"ContainerDied","Data":"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"}
Nov 02 22:25:14.513249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.513085    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng" event={"ID":"9f8970e7-a632-48d5-9f65-d7adf860c52c","Type":"ContainerDied","Data":"df08e7d160cfff0568a960dd9d9d32d2eabd76af21d8f83c4bfca7d8086fa98a"}
Nov 02 22:25:14.518850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.518812    4929 scope.go:122] "RemoveContainer" containerID="47c44f393eee673498bcbdf6e2e5784f31bf27579e0fbc1eeb342b88c6ddc3e2"
Nov 02 22:25:14.534930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.534907    4929 scope.go:122] "RemoveContainer" containerID="b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"
Nov 02 22:25:14.539509 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.539480    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:25:14.542947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.542923    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-419f68-55dddf9bdc-g8zct"]
Nov 02 22:25:14.544965 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.544946    4929 scope.go:122] "RemoveContainer" containerID="b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"
Nov 02 22:25:14.545239 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:14.545222    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0\": container with ID starting with b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0 not found: ID does not exist" containerID="b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"
Nov 02 22:25:14.545289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.545248    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0"} err="failed to get container status \"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0\": rpc error: code = NotFound desc = could not find container \"b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0\": container with ID starting with b6153eef3dc02b6e1409c35a9c5a68e0d8adf2ce41fcdb80a4f7dfea2761b5d0 not found: ID does not exist"
Nov 02 22:25:14.545289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.545265    4929 scope.go:122] "RemoveContainer" containerID="47573edb14c6dbb56412cc050cc9348b6ec0f42b4ba77d7a83c01bb07329ad0b"
Nov 02 22:25:14.553032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.553008    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:25:14.555874 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.555823    4929 scope.go:122] "RemoveContainer" containerID="f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"
Nov 02 22:25:14.556742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.556720    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-419f68-f944c6b85-76t78"]
Nov 02 22:25:14.567279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.567259    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:25:14.568810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.568787    4929 scope.go:122] "RemoveContainer" containerID="14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"
Nov 02 22:25:14.571482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.571456    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-c76zx"]
Nov 02 22:25:14.578345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.578322    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:25:14.579110 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.578966    4929 scope.go:122] "RemoveContainer" containerID="f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"
Nov 02 22:25:14.579680 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:14.579446    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7\": container with ID starting with f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7 not found: ID does not exist" containerID="f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"
Nov 02 22:25:14.579680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.579479    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7"} err="failed to get container status \"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7\": rpc error: code = NotFound desc = could not find container \"f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7\": container with ID starting with f71914c1d62a542168daf5f78823f78669d9e31ccf07ae7f45a3df34b729a3b7 not found: ID does not exist"
Nov 02 22:25:14.579680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.579519    4929 scope.go:122] "RemoveContainer" containerID="14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"
Nov 02 22:25:14.579913 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:14.579861    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840\": container with ID starting with 14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840 not found: ID does not exist" containerID="14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"
Nov 02 22:25:14.579913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.579892    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840"} err="failed to get container status \"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840\": rpc error: code = NotFound desc = could not find container \"14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840\": container with ID starting with 14f4fddafe9bea82d01a74203889d9737268f6081fd58cf44eafb01df1d75840 not found: ID does not exist"
Nov 02 22:25:14.580031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.579915    4929 scope.go:122] "RemoveContainer" containerID="a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"
Nov 02 22:25:14.583009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.582983    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-zvlng"]
Nov 02 22:25:14.590045 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.590026    4929 scope.go:122] "RemoveContainer" containerID="a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"
Nov 02 22:25:14.590322 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:14.590290    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d\": container with ID starting with a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d not found: ID does not exist" containerID="a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"
Nov 02 22:25:14.590412 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.590324    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d"} err="failed to get container status \"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d\": rpc error: code = NotFound desc = could not find container \"a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d\": container with ID starting with a0afef5f3c4f8d2d95375ad6f86b2a23373d7860fce0e28d5cbc68688b2b974d not found: ID does not exist"
Nov 02 22:25:14.617603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.615591    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:25:14.640290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.640244    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a54217/test-model-a54217-kserve-64c5b99579-z8dpn"]
Nov 02 22:25:14.646243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.645058    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:25:14.649052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.649026    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e49edd-7cbb84494d-lkr2h"]
Nov 02 22:25:14.805614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.805499    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" path="/var/lib/kubelet/pods/123aac54-2085-49f4-8e27-483ccf274a1e/volumes"
Nov 02 22:25:14.806050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.806035    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" path="/var/lib/kubelet/pods/354a47a7-8241-4d2a-a09c-dc75aaa12cd5/volumes"
Nov 02 22:25:14.806458 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.806445    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" path="/var/lib/kubelet/pods/48e8a816-11c1-4d18-8487-87097f71cb93/volumes"
Nov 02 22:25:14.807174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.807158    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" path="/var/lib/kubelet/pods/6a017c56-f362-43c3-9efd-3461e8dcb554/volumes"
Nov 02 22:25:14.807637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.807624    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" path="/var/lib/kubelet/pods/89759db7-7ede-4cd4-9cc0-7b08206cb781/volumes"
Nov 02 22:25:14.808115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:14.808100    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" path="/var/lib/kubelet/pods/9f8970e7-a632-48d5-9f65-d7adf860c52c/volumes"
Nov 02 22:25:15.676271 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:15.676240    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:25:15.866409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:15.866373    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:25:15.866798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:15.866593    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy" containerID="cri-o://a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4" gracePeriod=30
Nov 02 22:25:20.354112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.354066    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" podUID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" containerName="main" containerID="cri-o://6241bd78d3fb37a018a54adb638847161a88c271a6da3f627cae96b0c7957673" gracePeriod=45
Nov 02 22:25:20.577256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.565464    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:25:20.577256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.565751    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy" containerID="cri-o://29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70" gracePeriod=30
Nov 02 22:25:20.583582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.583365    4929 generic.go:356] "Generic (PLEG): container finished" podID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" containerID="6241bd78d3fb37a018a54adb638847161a88c271a6da3f627cae96b0c7957673" exitCode=0
Nov 02 22:25:20.583582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.583442    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" event={"ID":"69074b1c-17c2-4630-9889-7fb1c7b0a8d8","Type":"ContainerDied","Data":"6241bd78d3fb37a018a54adb638847161a88c271a6da3f627cae96b0c7957673"}
Nov 02 22:25:20.584895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.584866    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:25:20.586096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.585051    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy" containerID="cri-o://7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc" gracePeriod=30
Nov 02 22:25:20.694631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.691102    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing" containerID="cri-o://03b86023f7a6b25cda481d57e23d337ae83926a5edce978987331711d712a57d" gracePeriod=25
Nov 02 22:25:20.861084 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.861048    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:25:20.963236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963153    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963197    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963306    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\" (UniqueName: \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963372    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963507 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963426    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:20.963507 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963446    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963701 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963649    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\") pod \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\" (UID: \"69074b1c-17c2-4630-9889-7fb1c7b0a8d8\") "
Nov 02 22:25:20.963783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963701    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:20.963935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.963910    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:20.964605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.964578    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:20.964726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.964608    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:20.964726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.964623    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:20.965749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.965728    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:20.965840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.965823    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:20.965915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:20.965823    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp" pod "69074b1c-17c2-4630-9889-7fb1c7b0a8d8" (UID: "69074b1c-17c2-4630-9889-7fb1c7b0a8d8"). InnerVolumeSpecName "kube-api-access-ps4wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:21.066797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.066760    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.066972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.066800    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4wp\" (UniqueName: \"kubernetes.io/projected/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-kube-api-access-ps4wp\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.066972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.066818    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69074b1c-17c2-4630-9889-7fb1c7b0a8d8-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.141571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.141529    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:25:21.271673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.270142    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.271673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.270565    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:21.271673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.270720    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.271673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.271180    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.276997    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:21.277175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277159    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277313    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277365    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277411    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277443    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.277768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277484    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\") pod \"1560d752-71a3-4287-95e4-a6cf2a172c12\" (UID: \"1560d752-71a3-4287-95e4-a6cf2a172c12\") "
Nov 02 22:25:21.278807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277832    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:21.278807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.277844    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:21.278807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.278026    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:21.278807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.278418    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279173    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279216    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279240    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279295    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279309    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.279367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.279324    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1560d752-71a3-4287-95e4-a6cf2a172c12-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.280981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.280959    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:21.281751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.281727    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:21.282108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.282087    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545" pod "1560d752-71a3-4287-95e4-a6cf2a172c12" (UID: "1560d752-71a3-4287-95e4-a6cf2a172c12"). InnerVolumeSpecName "kube-api-access-b9545". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:21.381249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.381214    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9545\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-kube-api-access-b9545\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.381249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.381250    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.381648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.381264    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1560d752-71a3-4287-95e4-a6cf2a172c12-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:21.594195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.594105    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"
Nov 02 22:25:21.594373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.594108    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx" event={"ID":"69074b1c-17c2-4630-9889-7fb1c7b0a8d8","Type":"ContainerDied","Data":"093cb4e0b55c883bd81ec2a00c13992560c2cbab873d99d6370f77c7fd5aa30c"}
Nov 02 22:25:21.594373 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.594233    4929 scope.go:122] "RemoveContainer" containerID="6241bd78d3fb37a018a54adb638847161a88c271a6da3f627cae96b0c7957673"
Nov 02 22:25:21.597210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.597015    4929 generic.go:356] "Generic (PLEG): container finished" podID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerID="a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4" exitCode=0
Nov 02 22:25:21.597210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.597071    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerDied","Data":"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"}
Nov 02 22:25:21.597210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.597081    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"
Nov 02 22:25:21.597679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.597166    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2" event={"ID":"1560d752-71a3-4287-95e4-a6cf2a172c12","Type":"ContainerDied","Data":"c327abbcafca62afefdc1251cf0dc02acbcc7ac0517a5827e0afa060bb4a0d60"}
Nov 02 22:25:21.608035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.608017    4929 scope.go:122] "RemoveContainer" containerID="a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"
Nov 02 22:25:21.617369 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.617350    4929 scope.go:122] "RemoveContainer" containerID="2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"
Nov 02 22:25:21.623450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.623421    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:25:21.627872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.627853    4929 scope.go:122] "RemoveContainer" containerID="a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"
Nov 02 22:25:21.628206 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:21.628174    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4\": container with ID starting with a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4 not found: ID does not exist" containerID="a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"
Nov 02 22:25:21.628299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.628215    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4"} err="failed to get container status \"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4\": rpc error: code = NotFound desc = could not find container \"a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4\": container with ID starting with a2ca0a42c2991fb03ad2d187532335fc01eb1cf2fd1d4643bf39ad49f53f53c4 not found: ID does not exist"
Nov 02 22:25:21.628299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.628241    4929 scope.go:122] "RemoveContainer" containerID="2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"
Nov 02 22:25:21.628567 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:21.628523    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803\": container with ID starting with 2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803 not found: ID does not exist" containerID="2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"
Nov 02 22:25:21.628682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.628566    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803"} err="failed to get container status \"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803\": rpc error: code = NotFound desc = could not find container \"2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803\": container with ID starting with 2acc72a854a43d972cd959cde8b56507a0a86604d1a4926510e6ea407aa02803 not found: ID does not exist"
Nov 02 22:25:21.630576 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.630555    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-wjdt2"]
Nov 02 22:25:21.645775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.645753    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:25:21.648962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:21.648936    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-00c72a/test-model-00c72a-kserve-5f76bbbd45-cmfnx"]
Nov 02 22:25:22.806248 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:22.806218    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" path="/var/lib/kubelet/pods/1560d752-71a3-4287-95e4-a6cf2a172c12/volumes"
Nov 02 22:25:22.806876 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:22.806856    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" path="/var/lib/kubelet/pods/69074b1c-17c2-4630-9889-7fb1c7b0a8d8/volumes"
Nov 02 22:25:25.878030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.878004    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:25:25.927978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.927948    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:25:25.949520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949485    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949594    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949652    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949689    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949716    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949760    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949794    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949833    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949878    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.949988    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950021    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950054    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950164    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950221    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\") pod \"1c177830-557e-4f11-bc68-ba22db731fda\" (UID: \"1c177830-557e-4f11-bc68-ba22db731fda\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950277    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950335    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950404    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.951768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.950439    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\") pod \"c797e29a-64d2-4f98-b179-ee4e332e00df\" (UID: \"c797e29a-64d2-4f98-b179-ee4e332e00df\") "
Nov 02 22:25:25.954187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.953604    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.954187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.953739    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.954187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.953906    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.954187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.953993    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.954673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.954610    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.955565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.955099    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:25.955565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.955510    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:25.956019 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.955747    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.956206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.956141    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "kube-api-access-rsdms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:25.956206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.956164    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.956323 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.956232    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.957988 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.957948    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "kube-api-access-v2n6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:25.959316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.959284    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:25.959492 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.959344    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.959606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.959581    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:25.959981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.959957    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token" pod "c797e29a-64d2-4f98-b179-ee4e332e00df" (UID: "c797e29a-64d2-4f98-b179-ee4e332e00df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:25.960402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.960373    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:25.960501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:25.960416    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo" pod "1c177830-557e-4f11-bc68-ba22db731fda" (UID: "1c177830-557e-4f11-bc68-ba22db731fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:26.003938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.003893    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:26.052139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052104    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052134    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052139 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052145    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052159    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052169    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c177830-557e-4f11-bc68-ba22db731fda-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052178    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsdms\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-kube-api-access-rsdms\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052187    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052199    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c177830-557e-4f11-bc68-ba22db731fda-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052210    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c797e29a-64d2-4f98-b179-ee4e332e00df-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052223    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052236    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052248    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052261    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n6n\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-kube-api-access-v2n6n\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052271    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052280    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c797e29a-64d2-4f98-b179-ee4e332e00df-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052292    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c177830-557e-4f11-bc68-ba22db731fda-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052304    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.052399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.052313    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c177830-557e-4f11-bc68-ba22db731fda-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:26.636168 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.636131    4929 generic.go:356] "Generic (PLEG): container finished" podID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerID="7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc" exitCode=0
Nov 02 22:25:26.636309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.636174    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"
Nov 02 22:25:26.636309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.636214    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" event={"ID":"c797e29a-64d2-4f98-b179-ee4e332e00df","Type":"ContainerDied","Data":"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"}
Nov 02 22:25:26.636309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.636272    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth" event={"ID":"c797e29a-64d2-4f98-b179-ee4e332e00df","Type":"ContainerDied","Data":"38d9d6ed89fe541b56d2317dbd9a4cb65262f589ec09c9589876a3a3f6e9c12a"}
Nov 02 22:25:26.636309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.636304    4929 scope.go:122] "RemoveContainer" containerID="7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"
Nov 02 22:25:26.637967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.637942    4929 generic.go:356] "Generic (PLEG): container finished" podID="1c177830-557e-4f11-bc68-ba22db731fda" containerID="29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70" exitCode=0
Nov 02 22:25:26.638096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.637991    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"
Nov 02 22:25:26.638096 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.638066    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" event={"ID":"1c177830-557e-4f11-bc68-ba22db731fda","Type":"ContainerDied","Data":"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"}
Nov 02 22:25:26.638236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.638131    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg" event={"ID":"1c177830-557e-4f11-bc68-ba22db731fda","Type":"ContainerDied","Data":"cc8fadeb93e8ba06b902f5974557a93bd79548b36a896f718b09162f35a86a72"}
Nov 02 22:25:26.649309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.649285    4929 scope.go:122] "RemoveContainer" containerID="7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"
Nov 02 22:25:26.649649 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:26.649624    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc\": container with ID starting with 7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc not found: ID does not exist" containerID="7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"
Nov 02 22:25:26.649739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.649657    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc"} err="failed to get container status \"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc\": rpc error: code = NotFound desc = could not find container \"7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc\": container with ID starting with 7dba02a35faab9dd2561b893c13373a9041e85aafed96387175e73ed068f78fc not found: ID does not exist"
Nov 02 22:25:26.649739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.649678    4929 scope.go:122] "RemoveContainer" containerID="29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"
Nov 02 22:25:26.659671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.659650    4929 scope.go:122] "RemoveContainer" containerID="29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"
Nov 02 22:25:26.659970 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:26.659951    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70\": container with ID starting with 29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70 not found: ID does not exist" containerID="29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"
Nov 02 22:25:26.660061 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.659977    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70"} err="failed to get container status \"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70\": rpc error: code = NotFound desc = could not find container \"29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70\": container with ID starting with 29f4ce229dd784c3c73665e8e6a5c462811ba0b6c4b1fb92fc9356edd9737a70 not found: ID does not exist"
Nov 02 22:25:26.670158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.670132    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:25:26.673575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.673542    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-2bmcg"]
Nov 02 22:25:26.679345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.679324    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:25:26.682873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.682852    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-gldth"]
Nov 02 22:25:26.806307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.806275    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c177830-557e-4f11-bc68-ba22db731fda" path="/var/lib/kubelet/pods/1c177830-557e-4f11-bc68-ba22db731fda/volumes"
Nov 02 22:25:26.806908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:26.806890    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" path="/var/lib/kubelet/pods/c797e29a-64d2-4f98-b179-ee4e332e00df/volumes"
Nov 02 22:25:27.496727 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:27.496692    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:25:27.769413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:27.769307    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" podUID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" containerName="main" containerID="cri-o://da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796" gracePeriod=45
Nov 02 22:25:28.086217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.086188    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:25:28.087342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087321    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087476    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087563    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087642 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087608    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087665    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\" (UniqueName: \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087702    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\") pod \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\" (UID: \"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4\") "
Nov 02 22:25:28.087800 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087771    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:28.087833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087787    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:28.087833 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.087813    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:28.088218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.088194    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.088218 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.088219    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.088378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.088234    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.089950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.089928    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:28.090021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.089975    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "kube-api-access-hvtwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:28.090021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.089983    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs" pod "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" (UID: "b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:28.189429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.189394    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtwk\" (UniqueName: \"kubernetes.io/projected/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-kube-api-access-hvtwk\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.189429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.189426    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.189429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.189435    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:28.691140 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.691100    4929 generic.go:356] "Generic (PLEG): container finished" podID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" containerID="da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796" exitCode=0
Nov 02 22:25:28.691578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.691152    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"
Nov 02 22:25:28.691578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.691180    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" event={"ID":"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4","Type":"ContainerDied","Data":"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"}
Nov 02 22:25:28.691578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.691223    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6" event={"ID":"b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4","Type":"ContainerDied","Data":"bede19b396bf6cde3a2151d3c1416c447feafea3db5d91cbcc7237f5846639e0"}
Nov 02 22:25:28.691578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.691240    4929 scope.go:122] "RemoveContainer" containerID="da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"
Nov 02 22:25:28.701934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.701914    4929 scope.go:122] "RemoveContainer" containerID="da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"
Nov 02 22:25:28.702264 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:28.702244    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796\": container with ID starting with da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796 not found: ID does not exist" containerID="da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"
Nov 02 22:25:28.702315 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.702272    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796"} err="failed to get container status \"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796\": rpc error: code = NotFound desc = could not find container \"da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796\": container with ID starting with da7cac68f3758d7f4052e2fdffc632b0e54ace11546168bb4d0b03e222045796 not found: ID does not exist"
Nov 02 22:25:28.717742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.717526    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:25:28.720979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.720955    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a621f1/test-model-a621f1-kserve-5ff57648df-8grb6"]
Nov 02 22:25:28.805366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.805326    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" path="/var/lib/kubelet/pods/b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4/volumes"
Nov 02 22:25:28.977877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:28.977789    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:25:30.773523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:30.773484    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:25:30.774103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:30.773765    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" podUID="1310f3a9-262e-48fd-b6bf-703a20a036d3" containerName="maas-api" containerID="cri-o://061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c" gracePeriod=30
Nov 02 22:25:31.366032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:31.365998    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:25:31.376481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:31.376448    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:25:32.056686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.056659    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:25:32.152511 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.152474    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\") pod \"1310f3a9-262e-48fd-b6bf-703a20a036d3\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") "
Nov 02 22:25:32.152749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.152567    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\") pod \"1310f3a9-262e-48fd-b6bf-703a20a036d3\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") "
Nov 02 22:25:32.152749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.152706    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\" (UniqueName: \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\") pod \"1310f3a9-262e-48fd-b6bf-703a20a036d3\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") "
Nov 02 22:25:32.152873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.152778    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\") pod \"1310f3a9-262e-48fd-b6bf-703a20a036d3\" (UID: \"1310f3a9-262e-48fd-b6bf-703a20a036d3\") "
Nov 02 22:25:32.152972 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.152937    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca" pod "1310f3a9-262e-48fd-b6bf-703a20a036d3" (UID: "1310f3a9-262e-48fd-b6bf-703a20a036d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:32.153628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.153597    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1310f3a9-262e-48fd-b6bf-703a20a036d3-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:32.154902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.154875    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs" pod "1310f3a9-262e-48fd-b6bf-703a20a036d3" (UID: "1310f3a9-262e-48fd-b6bf-703a20a036d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:32.154996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.154933    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv" pod "1310f3a9-262e-48fd-b6bf-703a20a036d3" (UID: "1310f3a9-262e-48fd-b6bf-703a20a036d3"). InnerVolumeSpecName "kube-api-access-nwtbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:32.155126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.155107    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls" pod "1310f3a9-262e-48fd-b6bf-703a20a036d3" (UID: "1310f3a9-262e-48fd-b6bf-703a20a036d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:32.254694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.254661    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtbv\" (UniqueName: \"kubernetes.io/projected/1310f3a9-262e-48fd-b6bf-703a20a036d3-kube-api-access-nwtbv\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:32.254694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.254692    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:32.254694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.254702    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1310f3a9-262e-48fd-b6bf-703a20a036d3-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:32.743895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.743855    4929 generic.go:356] "Generic (PLEG): container finished" podID="1310f3a9-262e-48fd-b6bf-703a20a036d3" containerID="061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c" exitCode=0
Nov 02 22:25:32.744112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.743920    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"
Nov 02 22:25:32.744112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.743935    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" event={"ID":"1310f3a9-262e-48fd-b6bf-703a20a036d3","Type":"ContainerDied","Data":"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"}
Nov 02 22:25:32.744112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.744020    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl" event={"ID":"1310f3a9-262e-48fd-b6bf-703a20a036d3","Type":"ContainerDied","Data":"a7022ceb7d79d7ff79ec3bbe13456b8faff211db841ee5b62bb423f40f68f28e"}
Nov 02 22:25:32.744112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.744044    4929 scope.go:122] "RemoveContainer" containerID="061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"
Nov 02 22:25:32.756989 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.756969    4929 scope.go:122] "RemoveContainer" containerID="061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"
Nov 02 22:25:32.757260 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:32.757240    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c\": container with ID starting with 061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c not found: ID does not exist" containerID="061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"
Nov 02 22:25:32.757325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.757273    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c"} err="failed to get container status \"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c\": rpc error: code = NotFound desc = could not find container \"061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c\": container with ID starting with 061b6401e22fd32df3349e934b8ff977986b1361c5f201b579c8248743137f9c not found: ID does not exist"
Nov 02 22:25:32.769544 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.769503    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:25:32.774572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.774527    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a54217-55d597d6cb-ns5pl"]
Nov 02 22:25:32.828133 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:32.828086    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1310f3a9-262e-48fd-b6bf-703a20a036d3" path="/var/lib/kubelet/pods/1310f3a9-262e-48fd-b6bf-703a20a036d3/volumes"
Nov 02 22:25:34.876828 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:34.876790    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:25:34.877349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:34.877065    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy" containerID="cri-o://95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6" gracePeriod=30
Nov 02 22:25:36.004888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.004849    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:36.369181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.368646    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" podUID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" containerName="payload-processing" containerID="cri-o://62fd14e52480baa51dd624e2e385936f49280bf76dc3d77f52353044ad6fa64e" gracePeriod=25
Nov 02 22:25:36.394560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.384573    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" podUID="229f68f4-3a66-4375-b122-cfdf167fd1e9" containerName="payload-pre-processing" containerID="cri-o://189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d" gracePeriod=25
Nov 02 22:25:36.785777 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.785751    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:25:36.787980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.787949    4929 generic.go:356] "Generic (PLEG): container finished" podID="229f68f4-3a66-4375-b122-cfdf167fd1e9" containerID="189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d" exitCode=0
Nov 02 22:25:36.788128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.787998    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"
Nov 02 22:25:36.788128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.788022    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" event={"ID":"229f68f4-3a66-4375-b122-cfdf167fd1e9","Type":"ContainerDied","Data":"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"}
Nov 02 22:25:36.788128 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.788112    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr" event={"ID":"229f68f4-3a66-4375-b122-cfdf167fd1e9","Type":"ContainerDied","Data":"61fe169ec07d7190d7c885fe3a5325996f99c91d6d3dd795e2b98ba76aac7940"}
Nov 02 22:25:36.788296 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.788146    4929 scope.go:122] "RemoveContainer" containerID="189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"
Nov 02 22:25:36.796595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.796524    4929 generic.go:356] "Generic (PLEG): container finished" podID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" containerID="62fd14e52480baa51dd624e2e385936f49280bf76dc3d77f52353044ad6fa64e" exitCode=0
Nov 02 22:25:36.802617 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.802528    4929 scope.go:122] "RemoveContainer" containerID="189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"
Nov 02 22:25:36.802928 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:36.802900    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d\": container with ID starting with 189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d not found: ID does not exist" containerID="189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"
Nov 02 22:25:36.802997 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.802940    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d"} err="failed to get container status \"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d\": rpc error: code = NotFound desc = could not find container \"189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d\": container with ID starting with 189d8d9d20e6bf9633321f39f62236777efd39b8f58e38fd45bc355ba2fcc23d not found: ID does not exist"
Nov 02 22:25:36.814439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.814411    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" event={"ID":"6f36933c-5291-4a2b-9d3c-5ec3de657dd4","Type":"ContainerDied","Data":"62fd14e52480baa51dd624e2e385936f49280bf76dc3d77f52353044ad6fa64e"}
Nov 02 22:25:36.827075 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.827049    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\") pod \"229f68f4-3a66-4375-b122-cfdf167fd1e9\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") "
Nov 02 22:25:36.827207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.827111    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\" (UniqueName: \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\") pod \"229f68f4-3a66-4375-b122-cfdf167fd1e9\" (UID: \"229f68f4-3a66-4375-b122-cfdf167fd1e9\") "
Nov 02 22:25:36.827949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.827476    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume" pod "229f68f4-3a66-4375-b122-cfdf167fd1e9" (UID: "229f68f4-3a66-4375-b122-cfdf167fd1e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:36.827949 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.827920    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f68f4-3a66-4375-b122-cfdf167fd1e9-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:36.830025 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.829985    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj" pod "229f68f4-3a66-4375-b122-cfdf167fd1e9" (UID: "229f68f4-3a66-4375-b122-cfdf167fd1e9"). InnerVolumeSpecName "kube-api-access-hfstj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:36.836967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.836941    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:25:36.929385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.929285    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\") pod \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") "
Nov 02 22:25:36.929385 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.929381    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\" (UniqueName: \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\") pod \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\" (UID: \"6f36933c-5291-4a2b-9d3c-5ec3de657dd4\") "
Nov 02 22:25:36.930187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.930160    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume" pod "6f36933c-5291-4a2b-9d3c-5ec3de657dd4" (UID: "6f36933c-5291-4a2b-9d3c-5ec3de657dd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:36.933460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.933433    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4" pod "6f36933c-5291-4a2b-9d3c-5ec3de657dd4" (UID: "6f36933c-5291-4a2b-9d3c-5ec3de657dd4"). InnerVolumeSpecName "kube-api-access-fx7k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:36.937831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.937796    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:36.937943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.937833    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7k4\" (UniqueName: \"kubernetes.io/projected/6f36933c-5291-4a2b-9d3c-5ec3de657dd4-kube-api-access-fx7k4\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:36.937943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:36.937849    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfstj\" (UniqueName: \"kubernetes.io/projected/229f68f4-3a66-4375-b122-cfdf167fd1e9-kube-api-access-hfstj\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:37.112493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.112458    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:25:37.119891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.118715    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-df8835-79c47b7d7-b92hr"]
Nov 02 22:25:37.806871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.806793    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"
Nov 02 22:25:37.806871 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.806830    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7" event={"ID":"6f36933c-5291-4a2b-9d3c-5ec3de657dd4","Type":"ContainerDied","Data":"2956ad461df2f19738674022e65f87f984f2ca78eadf2d6bb4bc7af526e491ab"}
Nov 02 22:25:37.807091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.806897    4929 scope.go:122] "RemoveContainer" containerID="62fd14e52480baa51dd624e2e385936f49280bf76dc3d77f52353044ad6fa64e"
Nov 02 22:25:37.830880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.830847    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:25:37.833841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:37.833817    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-df8835-6c4bdcc7c7-9ncf7"]
Nov 02 22:25:38.784127 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:38.784086    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:25:38.784667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:38.784337    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy" containerID="cri-o://e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee" gracePeriod=30
Nov 02 22:25:38.806306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:38.806277    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229f68f4-3a66-4375-b122-cfdf167fd1e9" path="/var/lib/kubelet/pods/229f68f4-3a66-4375-b122-cfdf167fd1e9/volumes"
Nov 02 22:25:38.806906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:38.806881    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" path="/var/lib/kubelet/pods/6f36933c-5291-4a2b-9d3c-5ec3de657dd4/volumes"
Nov 02 22:25:39.396872 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:39.396835    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:25:40.159782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.159716    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:25:40.236619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236581    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.236619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236621    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.236873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236701    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.236873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236727    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.236873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236811    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.236873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236849    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.237102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.236886    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.237102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237003    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:40.237102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237014    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:40.237102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237045    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.237328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237154    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\") pod \"ed101f72-399c-4bae-999d-68f0b1995f3a\" (UID: \"ed101f72-399c-4bae-999d-68f0b1995f3a\") "
Nov 02 22:25:40.237392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237354    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:40.237449 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237284    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:40.237500 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237445    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:40.237895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237867    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.238029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237899    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.238029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237914    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed101f72-399c-4bae-999d-68f0b1995f3a-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.238029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237927    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.238029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.237943    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.238952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.238927    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:40.239098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.239071    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "kube-api-access-n4d9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:40.239211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.239159    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:40.239436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.239414    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo" pod "ed101f72-399c-4bae-999d-68f0b1995f3a" (UID: "ed101f72-399c-4bae-999d-68f0b1995f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:40.339294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.339256    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4d9v\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-kube-api-access-n4d9v\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.339294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.339290    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.339294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.339300    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.339573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.339309    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed101f72-399c-4bae-999d-68f0b1995f3a-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:40.835002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.834970    4929 generic.go:356] "Generic (PLEG): container finished" podID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerID="95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6" exitCode=0
Nov 02 22:25:40.835129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.835024    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"
Nov 02 22:25:40.835129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.835056    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerDied","Data":"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"}
Nov 02 22:25:40.835129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.835109    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5" event={"ID":"ed101f72-399c-4bae-999d-68f0b1995f3a","Type":"ContainerDied","Data":"99791ee724799d87023270b48c00253aaeb4b8139ea3ddbffd92b1e184e8e987"}
Nov 02 22:25:40.835306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.835140    4929 scope.go:122] "RemoveContainer" containerID="95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"
Nov 02 22:25:40.837832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.837803    4929 generic.go:356] "Generic (PLEG): container finished" podID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerID="03b86023f7a6b25cda481d57e23d337ae83926a5edce978987331711d712a57d" exitCode=0
Nov 02 22:25:40.837976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.837956    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" event={"ID":"616c0e90-878a-4090-8504-ecaea0c5d1d0","Type":"ContainerDied","Data":"03b86023f7a6b25cda481d57e23d337ae83926a5edce978987331711d712a57d"}
Nov 02 22:25:40.850400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.850377    4929 scope.go:122] "RemoveContainer" containerID="1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"
Nov 02 22:25:40.857979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.857936    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:25:40.862608 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.862525    4929 scope.go:122] "RemoveContainer" containerID="95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"
Nov 02 22:25:40.864159 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:40.864132    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6\": container with ID starting with 95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6 not found: ID does not exist" containerID="95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"
Nov 02 22:25:40.864269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.864166    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6"} err="failed to get container status \"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6\": rpc error: code = NotFound desc = could not find container \"95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6\": container with ID starting with 95a451e53d96c4be7da06218967f38fb78a6bbbee72b65fade6fab3bb29120a6 not found: ID does not exist"
Nov 02 22:25:40.864269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.864195    4929 scope.go:122] "RemoveContainer" containerID="1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"
Nov 02 22:25:40.864472 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:40.864454    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf\": container with ID starting with 1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf not found: ID does not exist" containerID="1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"
Nov 02 22:25:40.864522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.864476    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf"} err="failed to get container status \"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf\": rpc error: code = NotFound desc = could not find container \"1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf\": container with ID starting with 1ba28384fee153df1ef52b0120ee1e5b7e4da560e2fda3c26196ee138ba248bf not found: ID does not exist"
Nov 02 22:25:40.866487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.866461    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e30968-openshift-default-dcffbb757-nwrx5"]
Nov 02 22:25:40.991236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:40.991165    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:25:41.154414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.154380    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\") pod \"616c0e90-878a-4090-8504-ecaea0c5d1d0\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") "
Nov 02 22:25:41.154627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.154447    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\" (UniqueName: \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\") pod \"616c0e90-878a-4090-8504-ecaea0c5d1d0\" (UID: \"616c0e90-878a-4090-8504-ecaea0c5d1d0\") "
Nov 02 22:25:41.154797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.154774    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume" pod "616c0e90-878a-4090-8504-ecaea0c5d1d0" (UID: "616c0e90-878a-4090-8504-ecaea0c5d1d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:41.154970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.154957    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c0e90-878a-4090-8504-ecaea0c5d1d0-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:41.156662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.156635    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2" pod "616c0e90-878a-4090-8504-ecaea0c5d1d0" (UID: "616c0e90-878a-4090-8504-ecaea0c5d1d0"). InnerVolumeSpecName "kube-api-access-cswf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:41.256592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.256505    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswf2\" (UniqueName: \"kubernetes.io/projected/616c0e90-878a-4090-8504-ecaea0c5d1d0-kube-api-access-cswf2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:41.846814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.846783    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"
Nov 02 22:25:41.847004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.846812    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln" event={"ID":"616c0e90-878a-4090-8504-ecaea0c5d1d0","Type":"ContainerDied","Data":"4efe25a72e7433a1b3ed8f74f651be08c984cc001e80aa1a44f45714801a1941"}
Nov 02 22:25:41.847004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.846867    4929 scope.go:122] "RemoveContainer" containerID="03b86023f7a6b25cda481d57e23d337ae83926a5edce978987331711d712a57d"
Nov 02 22:25:41.870129 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.870094    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:25:41.874097 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:41.874072    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e30968-749646c859-88gln"]
Nov 02 22:25:42.806613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:42.806578    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" path="/var/lib/kubelet/pods/616c0e90-878a-4090-8504-ecaea0c5d1d0/volumes"
Nov 02 22:25:42.807068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:42.807053    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" path="/var/lib/kubelet/pods/ed101f72-399c-4bae-999d-68f0b1995f3a/volumes"
Nov 02 22:25:43.999083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:43.999019    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" podUID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" containerName="main" containerID="cri-o://db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e" gracePeriod=45
Nov 02 22:25:44.147927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.147892    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:25:44.266667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.266641    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:25:44.301229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301193    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301277    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301320    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301384    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301418    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301487    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.301713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301515    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.301713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301674    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.301834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.301779    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.302126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.302102    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.302866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.302837    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.302987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.302525    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.303047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303024    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\") pod \"d7af1b66-40b9-4956-80f9-dc1f86cef997\" (UID: \"d7af1b66-40b9-4956-80f9-dc1f86cef997\") "
Nov 02 22:25:44.303413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303387    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:44.303750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303725    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.303847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303754    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.303847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303769    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.303847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303784    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.303847 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.303799    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7af1b66-40b9-4956-80f9-dc1f86cef997-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.304274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.304250    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:44.304357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.304249    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:44.304463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.304449    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.306524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.306498    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl" pod "d7af1b66-40b9-4956-80f9-dc1f86cef997" (UID: "d7af1b66-40b9-4956-80f9-dc1f86cef997"). InnerVolumeSpecName "kube-api-access-ccvbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:44.397340 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.397300    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing" containerID="cri-o://ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72" gracePeriod=25
Nov 02 22:25:44.404194 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404166    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\" (UniqueName: \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404383    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404436    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404551    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404584    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404599    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.404712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404641    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\") pod \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\" (UID: \"99917624-eb73-4e3a-a1f3-b504df6ca6e1\") "
Nov 02 22:25:44.404712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404697    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.404827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.404811    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.405288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405266    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405292    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405309    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405323    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405337    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7af1b66-40b9-4956-80f9-dc1f86cef997-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405351    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.405396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.405368    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvbl\" (UniqueName: \"kubernetes.io/projected/d7af1b66-40b9-4956-80f9-dc1f86cef997-kube-api-access-ccvbl\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.406778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.406755    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:44.406962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.406939    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:44.407017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.406970    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh" pod "99917624-eb73-4e3a-a1f3-b504df6ca6e1" (UID: "99917624-eb73-4e3a-a1f3-b504df6ca6e1"). InnerVolumeSpecName "kube-api-access-kq2bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:44.506453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.506413    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99917624-eb73-4e3a-a1f3-b504df6ca6e1-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.506453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.506449    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99917624-eb73-4e3a-a1f3-b504df6ca6e1-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.506453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.506458    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq2bh\" (UniqueName: \"kubernetes.io/projected/99917624-eb73-4e3a-a1f3-b504df6ca6e1-kube-api-access-kq2bh\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:44.960294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.960251    4929 generic.go:356] "Generic (PLEG): container finished" podID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" containerID="db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e" exitCode=0
Nov 02 22:25:44.960466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.960355    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" event={"ID":"99917624-eb73-4e3a-a1f3-b504df6ca6e1","Type":"ContainerDied","Data":"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"}
Nov 02 22:25:44.960466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.960394    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78" event={"ID":"99917624-eb73-4e3a-a1f3-b504df6ca6e1","Type":"ContainerDied","Data":"177904fcbd74c29034e00fcadd3334882428013920e8716033d7db376a504a68"}
Nov 02 22:25:44.960466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.960416    4929 scope.go:122] "RemoveContainer" containerID="db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"
Nov 02 22:25:44.960709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.960587    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"
Nov 02 22:25:44.967182 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.967152    4929 generic.go:356] "Generic (PLEG): container finished" podID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerID="e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee" exitCode=0
Nov 02 22:25:44.967325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.967207    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" event={"ID":"d7af1b66-40b9-4956-80f9-dc1f86cef997","Type":"ContainerDied","Data":"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"}
Nov 02 22:25:44.967325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.967245    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7" event={"ID":"d7af1b66-40b9-4956-80f9-dc1f86cef997","Type":"ContainerDied","Data":"936eb5e8bf72f9fb8e639b486609bc87d1cb960d1b3d5685407a9de11e92af25"}
Nov 02 22:25:44.967325 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.967279    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"
Nov 02 22:25:44.976930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.976908    4929 scope.go:122] "RemoveContainer" containerID="db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"
Nov 02 22:25:44.977236 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:44.977216    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e\": container with ID starting with db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e not found: ID does not exist" containerID="db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"
Nov 02 22:25:44.977318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.977249    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e"} err="failed to get container status \"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e\": rpc error: code = NotFound desc = could not find container \"db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e\": container with ID starting with db05879df6d92d1f725f87eb11b65aef270ceca62b9fb2f89f9396b69615d36e not found: ID does not exist"
Nov 02 22:25:44.977318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.977273    4929 scope.go:122] "RemoveContainer" containerID="e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"
Nov 02 22:25:44.981891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.981866    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:25:44.985485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.985461    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5eba37/test-model-5eba37-kserve-55d476c4b9-wsl78"]
Nov 02 22:25:44.988724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.988707    4929 scope.go:122] "RemoveContainer" containerID="e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"
Nov 02 22:25:44.989002 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:44.988983    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee\": container with ID starting with e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee not found: ID does not exist" containerID="e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"
Nov 02 22:25:44.989056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.989012    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee"} err="failed to get container status \"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee\": rpc error: code = NotFound desc = could not find container \"e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee\": container with ID starting with e92fae489b58c45fee83dcd62f3f9c91a50d11497b16cb7403348ab053b44fee not found: ID does not exist"
Nov 02 22:25:44.996840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:44.996818    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:25:45.000484 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:45.000461    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-zk7l7"]
Nov 02 22:25:45.483903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:45.483864    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:25:45.497804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:45.497775    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:25:45.965283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:45.965233    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:25:45.965577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:45.965523    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" containerID="cri-o://86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342" gracePeriod=30
Nov 02 22:25:46.804969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:46.804935    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" path="/var/lib/kubelet/pods/99917624-eb73-4e3a-a1f3-b504df6ca6e1/volumes"
Nov 02 22:25:46.805501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:46.805485    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" path="/var/lib/kubelet/pods/d7af1b66-40b9-4956-80f9-dc1f86cef997/volumes"
Nov 02 22:25:46.868840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:46.868804    4929 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:25:46.868990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:46.868866    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:25:47.582247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:47.582205    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:48.620287 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:48.620250    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:25:48.620983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:48.620953    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy" containerID="cri-o://6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192" gracePeriod=30
Nov 02 22:25:49.403429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:49.403390    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:25:50.484490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:50.484446    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing" containerID="cri-o://6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac" gracePeriod=25
Nov 02 22:25:50.498264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:50.498213    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing" containerID="cri-o://d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20" gracePeriod=25
Nov 02 22:25:50.581282 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:50.581250    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:25:51.256755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.256473    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334091    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334221    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334287    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334377    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334432    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334494    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334670    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334803    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.334924    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\") pod \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\" (UID: \"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3\") "
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.336048    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:51.336568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.336369    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:51.343354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.342824    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:51.343354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.342836    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:51.343795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.343755    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:51.359554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.359497    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:51.359716 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.359663    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:51.359814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.359795    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "kube-api-access-p7zrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:51.364098 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.363988    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token" pod "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" (UID: "4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:51.436615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436568    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436612    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436624    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436637    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436646    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436655    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436663    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436672    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:51.436986 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:51.436680    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zrn\" (UniqueName: \"kubernetes.io/projected/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3-kube-api-access-p7zrn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:52.022431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.022390    4929 generic.go:356] "Generic (PLEG): container finished" podID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerID="86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342" exitCode=0
Nov 02 22:25:52.022903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.022446    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"
Nov 02 22:25:52.022903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.022467    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerDied","Data":"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"}
Nov 02 22:25:52.022903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.022522    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds" event={"ID":"4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3","Type":"ContainerDied","Data":"d7960f4e8f3bfc79c006477bb9bfcc1cf14a2962d365e77bbe2eb48d15306d45"}
Nov 02 22:25:52.022903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.022566    4929 scope.go:122] "RemoveContainer" containerID="86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"
Nov 02 22:25:52.034773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.034751    4929 scope.go:122] "RemoveContainer" containerID="7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"
Nov 02 22:25:52.050436 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.050412    4929 scope.go:122] "RemoveContainer" containerID="86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"
Nov 02 22:25:52.050775 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:52.050752    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342\": container with ID starting with 86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342 not found: ID does not exist" containerID="86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"
Nov 02 22:25:52.050865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.050782    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342"} err="failed to get container status \"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342\": rpc error: code = NotFound desc = could not find container \"86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342\": container with ID starting with 86f3f509498230c47306643223be048100e2b65ab84f5c23fb7bd05a7fa80342 not found: ID does not exist"
Nov 02 22:25:52.050865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.050802    4929 scope.go:122] "RemoveContainer" containerID="7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"
Nov 02 22:25:52.051063 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:52.051031    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe\": container with ID starting with 7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe not found: ID does not exist" containerID="7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"
Nov 02 22:25:52.051063 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.051055    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe"} err="failed to get container status \"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe\": rpc error: code = NotFound desc = could not find container \"7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe\": container with ID starting with 7438d56fcaa0b9e308ab1042190b104ccb306ab6491bf76e8552daad1cf754fe not found: ID does not exist"
Nov 02 22:25:52.052727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.052700    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:25:52.058616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.058579    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-f44ds"]
Nov 02 22:25:52.806309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:52.806275    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" path="/var/lib/kubelet/pods/4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3/volumes"
Nov 02 22:25:53.898179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.898118    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:25:53.972739 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972697    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.972940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972758    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.972940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972830    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.972940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972870    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.972940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972924    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.973191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.972981    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.973191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973022    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.973191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973024    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:53.973191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973045    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:53.973191 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973062    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.973448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973197    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\") pod \"3c0e783b-b508-4204-bd25-c411160af26e\" (UID: \"3c0e783b-b508-4204-bd25-c411160af26e\") "
Nov 02 22:25:53.973448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973259    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:53.973448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973304    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:53.973628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973575    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:53.973976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973829    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:53.973976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973856    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:53.973976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973870    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e783b-b508-4204-bd25-c411160af26e-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:53.973976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973883    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:53.973976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.973897    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:53.975947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.975908    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:53.976055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.975948    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "kube-api-access-db276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:53.976055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.976024    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:25:53.976176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:53.976053    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy" pod "3c0e783b-b508-4204-bd25-c411160af26e" (UID: "3c0e783b-b508-4204-bd25-c411160af26e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:25:54.075595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.075523    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:54.075595 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.075601    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e783b-b508-4204-bd25-c411160af26e-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:54.076681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.075630    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db276\" (UniqueName: \"kubernetes.io/projected/3c0e783b-b508-4204-bd25-c411160af26e-kube-api-access-db276\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:54.076681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.075645    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e783b-b508-4204-bd25-c411160af26e-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:54.076681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.076287    4929 generic.go:356] "Generic (PLEG): container finished" podID="3c0e783b-b508-4204-bd25-c411160af26e" containerID="6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192" exitCode=0
Nov 02 22:25:54.076681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.076405    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerDied","Data":"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"}
Nov 02 22:25:54.076681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.076411    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"
Nov 02 22:25:54.076928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.076822    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9" event={"ID":"3c0e783b-b508-4204-bd25-c411160af26e","Type":"ContainerDied","Data":"2b562a69a641bb8b37b9b2f61904496cd0e0b84d88422abd8e3c6ff7206eb7e6"}
Nov 02 22:25:54.076928 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.076859    4929 scope.go:122] "RemoveContainer" containerID="6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"
Nov 02 22:25:54.088467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.088440    4929 scope.go:122] "RemoveContainer" containerID="2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"
Nov 02 22:25:54.098166 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.098141    4929 scope.go:122] "RemoveContainer" containerID="6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"
Nov 02 22:25:54.098473 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:54.098451    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192\": container with ID starting with 6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192 not found: ID does not exist" containerID="6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"
Nov 02 22:25:54.098583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.098480    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192"} err="failed to get container status \"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192\": rpc error: code = NotFound desc = could not find container \"6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192\": container with ID starting with 6cadd37eb69fe2ecb781f5636b0cfa7d44a89a1e9df0280c95cf420f428bb192 not found: ID does not exist"
Nov 02 22:25:54.098583 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.098498    4929 scope.go:122] "RemoveContainer" containerID="2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"
Nov 02 22:25:54.098798 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:54.098782    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895\": container with ID starting with 2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895 not found: ID does not exist" containerID="2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"
Nov 02 22:25:54.098839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.098805    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895"} err="failed to get container status \"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895\": rpc error: code = NotFound desc = could not find container \"2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895\": container with ID starting with 2756d8cb2a50804a67849dcd8fc22abbe5ef89f54d63cff97af51b2fe4429895 not found: ID does not exist"
Nov 02 22:25:54.100396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.100377    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:25:54.104639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.104616    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-44mg9"]
Nov 02 22:25:54.404569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.404509    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing" containerID="cri-o://db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436" gracePeriod=25
Nov 02 22:25:54.706765 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.706669    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:54.805821 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:54.805789    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0e783b-b508-4204-bd25-c411160af26e" path="/var/lib/kubelet/pods/3c0e783b-b508-4204-bd25-c411160af26e/volumes"
Nov 02 22:25:55.338911 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:55.338861    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:25:55.350246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:55.350211    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:25:55.998685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:55.998647    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:57.036619 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:57.036581    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:25:57.196518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:57.196482    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:25:57.196778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:57.196749    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy" containerID="cri-o://4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f" gracePeriod=30
Nov 02 22:25:57.496831 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:25:57.496794    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:25:57.582500 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:25:57.582462    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:00.217584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.217549    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:26:00.217991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.217802    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy" containerID="cri-o://e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366" gracePeriod=30
Nov 02 22:26:00.360555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.355807    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerName="payload-pre-processing" containerID="cri-o://e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece" gracePeriod=25
Nov 02 22:26:00.380143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.374647    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" containerName="payload-processing" containerID="cri-o://18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e" gracePeriod=25
Nov 02 22:26:00.648968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.648938    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:26:00.700724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.700574    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:26:00.777001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.776815    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\" (UniqueName: \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\") pod \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") "
Nov 02 22:26:00.777220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.777190    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\") pod \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\" (UID: \"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc\") "
Nov 02 22:26:00.777466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.777440    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\" (UniqueName: \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\") pod \"39465d60-8f30-436d-b5df-e7d25adbca88\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") "
Nov 02 22:26:00.777559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.777510    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\") pod \"39465d60-8f30-436d-b5df-e7d25adbca88\" (UID: \"39465d60-8f30-436d-b5df-e7d25adbca88\") "
Nov 02 22:26:00.777612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.777582    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume" pod "dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" (UID: "dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:00.777971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.777945    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume" pod "39465d60-8f30-436d-b5df-e7d25adbca88" (UID: "39465d60-8f30-436d-b5df-e7d25adbca88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:00.778131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.778115    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39465d60-8f30-436d-b5df-e7d25adbca88-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:00.778184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.778138    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:00.779245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.779223    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2" pod "dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" (UID: "dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc"). InnerVolumeSpecName "kube-api-access-d4tg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:00.779565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.779529    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz" pod "39465d60-8f30-436d-b5df-e7d25adbca88" (UID: "39465d60-8f30-436d-b5df-e7d25adbca88"). InnerVolumeSpecName "kube-api-access-6qfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:00.880006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.879930    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfrz\" (UniqueName: \"kubernetes.io/projected/39465d60-8f30-436d-b5df-e7d25adbca88-kube-api-access-6qfrz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:00.880006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:00.879962    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tg2\" (UniqueName: \"kubernetes.io/projected/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc-kube-api-access-d4tg2\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:01.123474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.123441    4929 generic.go:356] "Generic (PLEG): container finished" podID="39465d60-8f30-436d-b5df-e7d25adbca88" containerID="18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e" exitCode=0
Nov 02 22:26:01.123680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.123488    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"
Nov 02 22:26:01.123680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.123522    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" event={"ID":"39465d60-8f30-436d-b5df-e7d25adbca88","Type":"ContainerDied","Data":"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"}
Nov 02 22:26:01.123680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.123607    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl" event={"ID":"39465d60-8f30-436d-b5df-e7d25adbca88","Type":"ContainerDied","Data":"895cf1ee17ee81734887d36c03ff5ec18ae064194c5354ec916f313fec8d0ede"}
Nov 02 22:26:01.123680 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.123638    4929 scope.go:122] "RemoveContainer" containerID="18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"
Nov 02 22:26:01.124991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.124969    4929 generic.go:356] "Generic (PLEG): container finished" podID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerID="e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece" exitCode=0
Nov 02 22:26:01.125101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.125087    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" event={"ID":"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc","Type":"ContainerDied","Data":"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"}
Nov 02 22:26:01.125154 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.125121    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j" event={"ID":"dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc","Type":"ContainerDied","Data":"bbe7c467f2ad18f28a473a76c330ecd7b2ed45d546d36e2191f73f07a3b81c01"}
Nov 02 22:26:01.125203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.125173    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"
Nov 02 22:26:01.134035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.134016    4929 scope.go:122] "RemoveContainer" containerID="18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"
Nov 02 22:26:01.134355 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:01.134332    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e\": container with ID starting with 18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e not found: ID does not exist" containerID="18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"
Nov 02 22:26:01.134451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.134360    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e"} err="failed to get container status \"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e\": rpc error: code = NotFound desc = could not find container \"18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e\": container with ID starting with 18f9be9620332e415afe90c6d9eb01205699a5cf1ab75e279b00ea9f507cb89e not found: ID does not exist"
Nov 02 22:26:01.134451 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.134376    4929 scope.go:122] "RemoveContainer" containerID="e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"
Nov 02 22:26:01.142895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.142868    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:26:01.143661 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.143651    4929 scope.go:122] "RemoveContainer" containerID="e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"
Nov 02 22:26:01.144043 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:01.144013    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece\": container with ID starting with e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece not found: ID does not exist" containerID="e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"
Nov 02 22:26:01.144471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.144071    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece"} err="failed to get container status \"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece\": rpc error: code = NotFound desc = could not find container \"e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece\": container with ID starting with e39e950d6d887bfc49505d6cc7a9a4678c1409886ad9c24d5bc4b1b4bc747ece not found: ID does not exist"
Nov 02 22:26:01.148275 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.148250    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7bb56e-6f4d55889f-zxf2j"]
Nov 02 22:26:01.157101 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.157082    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:26:01.162207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:01.162183    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7bb56e-6c5b45f5f7-hm5pl"]
Nov 02 22:26:02.467227 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.467170    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:26:02.600338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600307    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600365    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600409    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600499    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600594    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600691    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600729    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600737    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:02.600915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600766    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600831    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\") pod \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\" (UID: \"15a67bb9-b21b-4a9f-ad9a-73704cae7dab\") "
Nov 02 22:26:02.600915 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.600843    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:02.601130 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601103    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:02.601264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601177    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:02.601264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601205    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:02.601435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601419    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.601521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601436    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.601521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601445    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.601521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601454    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.601521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.601465    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.602933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.602903    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:02.603021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.602927    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:02.603184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.603169    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:02.603228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.603198    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn" pod "15a67bb9-b21b-4a9f-ad9a-73704cae7dab" (UID: "15a67bb9-b21b-4a9f-ad9a-73704cae7dab"). InnerVolumeSpecName "kube-api-access-xdzgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:02.634242 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.634213    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:26:02.703001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.702969    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.703001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.702999    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzgn\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-kube-api-access-xdzgn\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.703001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.703010    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.703230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.703019    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a67bb9-b21b-4a9f-ad9a-73704cae7dab-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:02.806114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.806038    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" path="/var/lib/kubelet/pods/39465d60-8f30-436d-b5df-e7d25adbca88/volumes"
Nov 02 22:26:02.806464 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:02.806448    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" path="/var/lib/kubelet/pods/dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc/volumes"
Nov 02 22:26:03.141328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.141292    4929 generic.go:356] "Generic (PLEG): container finished" podID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerID="4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f" exitCode=0
Nov 02 22:26:03.141553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.141350    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerDied","Data":"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"}
Nov 02 22:26:03.141553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.141369    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"
Nov 02 22:26:03.141553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.141404    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg" event={"ID":"15a67bb9-b21b-4a9f-ad9a-73704cae7dab","Type":"ContainerDied","Data":"beddc2aa941a543e1ab3485e99430b1f3fd9a596432dadc5729be3590392700e"}
Nov 02 22:26:03.141553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.141432    4929 scope.go:122] "RemoveContainer" containerID="4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"
Nov 02 22:26:03.151501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.151478    4929 scope.go:122] "RemoveContainer" containerID="a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"
Nov 02 22:26:03.160002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.159978    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:26:03.161671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.161622    4929 scope.go:122] "RemoveContainer" containerID="4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"
Nov 02 22:26:03.161971 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:03.161949    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f\": container with ID starting with 4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f not found: ID does not exist" containerID="4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"
Nov 02 22:26:03.162067 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.161977    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f"} err="failed to get container status \"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f\": rpc error: code = NotFound desc = could not find container \"4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f\": container with ID starting with 4fa5dba31841af84a861cfd5eb530f6dbe90c5e6b1d8e48eba8b40d5a5ec7b2f not found: ID does not exist"
Nov 02 22:26:03.162067 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.161998    4929 scope.go:122] "RemoveContainer" containerID="a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"
Nov 02 22:26:03.162237 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:03.162217    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16\": container with ID starting with a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16 not found: ID does not exist" containerID="a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"
Nov 02 22:26:03.162301 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.162242    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16"} err="failed to get container status \"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16\": rpc error: code = NotFound desc = could not find container \"a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16\": container with ID starting with a8528eb0872a0f253503c6825f976816df9d5c8b1961d5030717272ec50f1b16 not found: ID does not exist"
Nov 02 22:26:03.163545 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:03.163513    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-2v2zg"]
Nov 02 22:26:04.666663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.666597    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:26:04.707267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.707227    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:04.806652 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.806619    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" path="/var/lib/kubelet/pods/15a67bb9-b21b-4a9f-ad9a-73704cae7dab/volumes"
Nov 02 22:26:04.832869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.832836    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\") pod \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") "
Nov 02 22:26:04.833033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.832977    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\" (UniqueName: \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\") pod \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\" (UID: \"91a9d4a4-eaed-4349-8b97-1f25f09c6444\") "
Nov 02 22:26:04.833725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.833290    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume" pod "91a9d4a4-eaed-4349-8b97-1f25f09c6444" (UID: "91a9d4a4-eaed-4349-8b97-1f25f09c6444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:04.833887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.833865    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9d4a4-eaed-4349-8b97-1f25f09c6444-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:04.835810 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.835787    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v" pod "91a9d4a4-eaed-4349-8b97-1f25f09c6444" (UID: "91a9d4a4-eaed-4349-8b97-1f25f09c6444"). InnerVolumeSpecName "kube-api-access-9t82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:04.935413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:04.935324    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t82v\" (UniqueName: \"kubernetes.io/projected/91a9d4a4-eaed-4349-8b97-1f25f09c6444-kube-api-access-9t82v\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.156085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.156044    4929 generic.go:356] "Generic (PLEG): container finished" podID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerID="ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72" exitCode=0
Nov 02 22:26:05.156313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.156095    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"
Nov 02 22:26:05.156313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.156108    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" event={"ID":"91a9d4a4-eaed-4349-8b97-1f25f09c6444","Type":"ContainerDied","Data":"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"}
Nov 02 22:26:05.156313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.156178    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf" event={"ID":"91a9d4a4-eaed-4349-8b97-1f25f09c6444","Type":"ContainerDied","Data":"fd34b6ecca5844b1d558a5fea2d9dfa09769fd2c72fb54978dc3382f222dc3b0"}
Nov 02 22:26:05.156313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.156212    4929 scope.go:122] "RemoveContainer" containerID="ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"
Nov 02 22:26:05.166356 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.166338    4929 scope.go:122] "RemoveContainer" containerID="ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"
Nov 02 22:26:05.166675 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:05.166655    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72\": container with ID starting with ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72 not found: ID does not exist" containerID="ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"
Nov 02 22:26:05.166735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.166684    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72"} err="failed to get container status \"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72\": rpc error: code = NotFound desc = could not find container \"ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72\": container with ID starting with ccab5427afef34282586d9fb484abace23ed225beb75f17783378b3fc1d20f72 not found: ID does not exist"
Nov 02 22:26:05.177860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.177834    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:26:05.184475 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.184452    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d28bd1-779b655879-jsxjf"]
Nov 02 22:26:05.520368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.520292    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:26:05.642893 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.642861    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.642930    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.642966    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643007    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643053    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643137    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643211    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643247    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643298    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:05.643729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643306    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:05.643729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643346    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\") pod \"8637bbe2-fa8d-48ef-a592-b131944fed5f\" (UID: \"8637bbe2-fa8d-48ef-a592-b131944fed5f\") "
Nov 02 22:26:05.643729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643501    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:05.643896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643605    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:05.643896 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643626    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:05.644000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.643986    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637bbe2-fa8d-48ef-a592-b131944fed5f-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.644055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.644007    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.644055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.644025    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.644055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.644040    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.644170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.644054    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.645359 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.645336    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:05.645480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.645446    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:05.645702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.645675    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:05.645702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.645700    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh" pod "8637bbe2-fa8d-48ef-a592-b131944fed5f" (UID: "8637bbe2-fa8d-48ef-a592-b131944fed5f"). InnerVolumeSpecName "kube-api-access-b5vfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:05.745350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.745309    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.745350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.745342    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vfh\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-kube-api-access-b5vfh\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.745350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.745355    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.745846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.745364    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637bbe2-fa8d-48ef-a592-b131944fed5f-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:05.999151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:05.999105    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:06.163807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.163771    4929 generic.go:356] "Generic (PLEG): container finished" podID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerID="e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366" exitCode=0
Nov 02 22:26:06.164013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.163852    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"
Nov 02 22:26:06.164013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.163854    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerDied","Data":"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"}
Nov 02 22:26:06.164013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.163945    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6" event={"ID":"8637bbe2-fa8d-48ef-a592-b131944fed5f","Type":"ContainerDied","Data":"81393df1e6cff0ca9ea1129e1684540bfe9a3b0988563074432e4f4b4a87305d"}
Nov 02 22:26:06.164013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.163960    4929 scope.go:122] "RemoveContainer" containerID="e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"
Nov 02 22:26:06.177087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.176955    4929 scope.go:122] "RemoveContainer" containerID="5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"
Nov 02 22:26:06.188141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.188114    4929 scope.go:122] "RemoveContainer" containerID="e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"
Nov 02 22:26:06.188445 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:06.188425    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366\": container with ID starting with e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366 not found: ID does not exist" containerID="e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"
Nov 02 22:26:06.188512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.188455    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366"} err="failed to get container status \"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366\": rpc error: code = NotFound desc = could not find container \"e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366\": container with ID starting with e43ee681307e9b79d62a9805231f465df92234c3e912adf4fdc723e81749f366 not found: ID does not exist"
Nov 02 22:26:06.188512 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.188473    4929 scope.go:122] "RemoveContainer" containerID="5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"
Nov 02 22:26:06.189718 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:06.188878    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a\": container with ID starting with 5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a not found: ID does not exist" containerID="5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"
Nov 02 22:26:06.189718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.188912    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a"} err="failed to get container status \"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a\": rpc error: code = NotFound desc = could not find container \"5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a\": container with ID starting with 5d89a0ac9325c709e998dc39afc8ba4f097609f0ba42d2d96308c7179b88772a not found: ID does not exist"
Nov 02 22:26:06.192343 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.192320    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:26:06.194889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.194868    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a621f1-openshift-default-6c668c4ddc-vvwx6"]
Nov 02 22:26:06.804783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.804749    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" path="/var/lib/kubelet/pods/8637bbe2-fa8d-48ef-a592-b131944fed5f/volumes"
Nov 02 22:26:06.805230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:06.805216    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" path="/var/lib/kubelet/pods/91a9d4a4-eaed-4349-8b97-1f25f09c6444/volumes"
Nov 02 22:26:07.036952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:07.036916    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:08.053977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:08.053945    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:26:08.379967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:08.379930    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:26:10.298208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.298175    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:26:10.315324 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.315296    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:26:10.774717 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.774686    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:26:10.797443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.797418    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:26:10.836586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.836476    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\" (UniqueName: \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\") pod \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") "
Nov 02 22:26:10.836738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.836622    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\" (UniqueName: \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\") pod \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") "
Nov 02 22:26:10.836738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.836697    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\") pod \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\" (UID: \"ed8f5f30-52a2-4af0-9375-128cac11fe8e\") "
Nov 02 22:26:10.836859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.836823    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\") pod \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\" (UID: \"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7\") "
Nov 02 22:26:10.839998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.839968    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume" pod "2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" (UID: "2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:10.841161 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.841132    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq" pod "ed8f5f30-52a2-4af0-9375-128cac11fe8e" (UID: "ed8f5f30-52a2-4af0-9375-128cac11fe8e"). InnerVolumeSpecName "kube-api-access-s88jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:10.844108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.843318    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg" pod "2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" (UID: "2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7"). InnerVolumeSpecName "kube-api-access-x67fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:10.844108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.843714    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume" pod "ed8f5f30-52a2-4af0-9375-128cac11fe8e" (UID: "ed8f5f30-52a2-4af0-9375-128cac11fe8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:10.935432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.935117    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:26:10.957555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.948378    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:26:10.975072 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.958504    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:10.975352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.975324    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88jq\" (UniqueName: \"kubernetes.io/projected/ed8f5f30-52a2-4af0-9375-128cac11fe8e-kube-api-access-s88jq\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:10.975466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.975452    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67fg\" (UniqueName: \"kubernetes.io/projected/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7-kube-api-access-x67fg\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:10.975578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:10.975564    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f5f30-52a2-4af0-9375-128cac11fe8e-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:11.201496 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.201464    4929 generic.go:356] "Generic (PLEG): container finished" podID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerID="6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac" exitCode=0
Nov 02 22:26:11.201738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.201523    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"
Nov 02 22:26:11.201738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.201569    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" event={"ID":"ed8f5f30-52a2-4af0-9375-128cac11fe8e","Type":"ContainerDied","Data":"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"}
Nov 02 22:26:11.201738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.201659    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t" event={"ID":"ed8f5f30-52a2-4af0-9375-128cac11fe8e","Type":"ContainerDied","Data":"a11ceaf33b94f5eaa2487b5ca1c652040985bed37803b282c16d9379ad5d92d9"}
Nov 02 22:26:11.201738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.201694    4929 scope.go:122] "RemoveContainer" containerID="6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"
Nov 02 22:26:11.203094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.203071    4929 generic.go:356] "Generic (PLEG): container finished" podID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerID="d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20" exitCode=0
Nov 02 22:26:11.203196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.203128    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"
Nov 02 22:26:11.203196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.203128    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" event={"ID":"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7","Type":"ContainerDied","Data":"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"}
Nov 02 22:26:11.203196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.203183    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq" event={"ID":"2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7","Type":"ContainerDied","Data":"49e84bb8654bf5703e4adbcd4094eb4ae77569c53e59f017cc418c1089945fec"}
Nov 02 22:26:11.211666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.211650    4929 scope.go:122] "RemoveContainer" containerID="6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"
Nov 02 22:26:11.211907 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:11.211891    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac\": container with ID starting with 6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac not found: ID does not exist" containerID="6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"
Nov 02 22:26:11.211960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.211915    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac"} err="failed to get container status \"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac\": rpc error: code = NotFound desc = could not find container \"6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac\": container with ID starting with 6a72200eb4ddcc16aac79356c7074e0a05308cf670c8607b47f5b99f653ec1ac not found: ID does not exist"
Nov 02 22:26:11.211960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.211931    4929 scope.go:122] "RemoveContainer" containerID="d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"
Nov 02 22:26:11.221616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.221591    4929 scope.go:122] "RemoveContainer" containerID="d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"
Nov 02 22:26:11.221916 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.221898    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:26:11.221996 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:11.221918    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20\": container with ID starting with d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20 not found: ID does not exist" containerID="d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"
Nov 02 22:26:11.221996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.221958    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20"} err="failed to get container status \"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20\": rpc error: code = NotFound desc = could not find container \"d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20\": container with ID starting with d5d896b8a1b255be75e62edc4832c5f4ce0f78c97e2adf2ed2faf89e7a69bc20 not found: ID does not exist"
Nov 02 22:26:11.228255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.228230    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00c72a-7df578bd47-jcqdq"]
Nov 02 22:26:11.252815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.252779    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:26:11.279555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:11.261515    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00c72a-5987b79b4f-k5t8t"]
Nov 02 22:26:12.805070 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:12.805039    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" path="/var/lib/kubelet/pods/2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7/volumes"
Nov 02 22:26:12.805445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:12.805431    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" path="/var/lib/kubelet/pods/ed8f5f30-52a2-4af0-9375-128cac11fe8e/volumes"
Nov 02 22:26:13.055150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:13.055063    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing" containerID="cri-o://4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d" gracePeriod=25
Nov 02 22:26:14.665750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.665686    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:26:14.728297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.728262    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\" (UniqueName: \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\") pod \"7a1525ec-7195-492c-a67c-fae208e3c824\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") "
Nov 02 22:26:14.728468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.728427    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\") pod \"7a1525ec-7195-492c-a67c-fae208e3c824\" (UID: \"7a1525ec-7195-492c-a67c-fae208e3c824\") "
Nov 02 22:26:14.728805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.728781    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume" pod "7a1525ec-7195-492c-a67c-fae208e3c824" (UID: "7a1525ec-7195-492c-a67c-fae208e3c824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:14.730383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.730361    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm" pod "7a1525ec-7195-492c-a67c-fae208e3c824" (UID: "7a1525ec-7195-492c-a67c-fae208e3c824"). InnerVolumeSpecName "kube-api-access-szxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:14.830353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.830323    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxwm\" (UniqueName: \"kubernetes.io/projected/7a1525ec-7195-492c-a67c-fae208e3c824-kube-api-access-szxwm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:14.830353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:14.830351    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1525ec-7195-492c-a67c-fae208e3c824-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:15.229708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.229642    4929 generic.go:356] "Generic (PLEG): container finished" podID="7a1525ec-7195-492c-a67c-fae208e3c824" containerID="db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436" exitCode=0
Nov 02 22:26:15.229897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.229714    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"
Nov 02 22:26:15.229897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.229728    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" event={"ID":"7a1525ec-7195-492c-a67c-fae208e3c824","Type":"ContainerDied","Data":"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"}
Nov 02 22:26:15.229897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.229816    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl" event={"ID":"7a1525ec-7195-492c-a67c-fae208e3c824","Type":"ContainerDied","Data":"d8dd2a97740e7c96ce24cad868ea1302d48b31d91ea5ea90a119055b681672ed"}
Nov 02 22:26:15.229897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.229844    4929 scope.go:122] "RemoveContainer" containerID="db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"
Nov 02 22:26:15.240047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.240028    4929 scope.go:122] "RemoveContainer" containerID="db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"
Nov 02 22:26:15.240319 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:15.240299    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436\": container with ID starting with db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436 not found: ID does not exist" containerID="db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"
Nov 02 22:26:15.240366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.240329    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436"} err="failed to get container status \"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436\": rpc error: code = NotFound desc = could not find container \"db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436\": container with ID starting with db819c9cb255d0e3e7338778f706eaa8fc2c76cc7b5023c67ea59bcf25448436 not found: ID does not exist"
Nov 02 22:26:15.247522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.247494    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:26:15.250046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.250022    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a621f1-5586d767b7-s5ckl"]
Nov 02 22:26:15.644945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.644912    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:26:15.937637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:15.936171    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" podUID="f20930ed-6381-4551-8423-e9a0abfab58d" containerName="payload-processing" containerID="cri-o://e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17" gracePeriod=25
Nov 02 22:26:16.193106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.193048    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:26:16.204295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.204262    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" podUID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" containerName="payload-pre-processing" containerID="cri-o://5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3" gracePeriod=25
Nov 02 22:26:16.238280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.238250    4929 generic.go:356] "Generic (PLEG): container finished" podID="f20930ed-6381-4551-8423-e9a0abfab58d" containerID="e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17" exitCode=0
Nov 02 22:26:16.238447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.238301    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"
Nov 02 22:26:16.238447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.238335    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" event={"ID":"f20930ed-6381-4551-8423-e9a0abfab58d","Type":"ContainerDied","Data":"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"}
Nov 02 22:26:16.238447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.238400    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb" event={"ID":"f20930ed-6381-4551-8423-e9a0abfab58d","Type":"ContainerDied","Data":"e3e43f6c3cf1af2420ffccbda061ad0c85ef4bc656f6863ca7abb5bc52fdd8c4"}
Nov 02 22:26:16.238447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.238431    4929 scope.go:122] "RemoveContainer" containerID="e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"
Nov 02 22:26:16.302463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.302426    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:26:16.316999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.316973    4929 scope.go:122] "RemoveContainer" containerID="e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"
Nov 02 22:26:16.317379 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:16.317355    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17\": container with ID starting with e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17 not found: ID does not exist" containerID="e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"
Nov 02 22:26:16.317450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.317387    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17"} err="failed to get container status \"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17\": rpc error: code = NotFound desc = could not find container \"e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17\": container with ID starting with e5b8f0a544bc06e3fd256473634a01da8aa842bfdb1538023b6b7dfe04baef17 not found: ID does not exist"
Nov 02 22:26:16.329321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.329295    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:26:16.352236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.352186    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\") pod \"f20930ed-6381-4551-8423-e9a0abfab58d\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") "
Nov 02 22:26:16.352708 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.352366    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\" (UniqueName: \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\") pod \"f20930ed-6381-4551-8423-e9a0abfab58d\" (UID: \"f20930ed-6381-4551-8423-e9a0abfab58d\") "
Nov 02 22:26:16.353663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.353616    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume" pod "f20930ed-6381-4551-8423-e9a0abfab58d" (UID: "f20930ed-6381-4551-8423-e9a0abfab58d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:16.354733 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.354710    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv" pod "f20930ed-6381-4551-8423-e9a0abfab58d" (UID: "f20930ed-6381-4551-8423-e9a0abfab58d"). InnerVolumeSpecName "kube-api-access-pzxjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:16.454299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.454270    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxjv\" (UniqueName: \"kubernetes.io/projected/f20930ed-6381-4551-8423-e9a0abfab58d-kube-api-access-pzxjv\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:16.454408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.454301    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20930ed-6381-4551-8423-e9a0abfab58d-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:16.459835 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.459811    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:26:16.555188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.555157    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\" (UniqueName: \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\") pod \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") "
Nov 02 22:26:16.555335 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.555298    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\") pod \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\" (UID: \"21e473fb-93fc-4e27-9d9e-241ea3ea9b60\") "
Nov 02 22:26:16.555672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.555641    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume" pod "21e473fb-93fc-4e27-9d9e-241ea3ea9b60" (UID: "21e473fb-93fc-4e27-9d9e-241ea3ea9b60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:16.557121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.557103    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88" pod "21e473fb-93fc-4e27-9d9e-241ea3ea9b60" (UID: "21e473fb-93fc-4e27-9d9e-241ea3ea9b60"). InnerVolumeSpecName "kube-api-access-lnf88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:16.560906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.560884    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:26:16.565724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.565702    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ac0910-789b646687-x27kb"]
Nov 02 22:26:16.656400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.656365    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnf88\" (UniqueName: \"kubernetes.io/projected/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-kube-api-access-lnf88\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:16.656400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.656392    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e473fb-93fc-4e27-9d9e-241ea3ea9b60-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:16.804481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.804402    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" path="/var/lib/kubelet/pods/7a1525ec-7195-492c-a67c-fae208e3c824/volumes"
Nov 02 22:26:16.804870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:16.804856    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20930ed-6381-4551-8423-e9a0abfab58d" path="/var/lib/kubelet/pods/f20930ed-6381-4551-8423-e9a0abfab58d/volumes"
Nov 02 22:26:17.246980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.246945    4929 generic.go:356] "Generic (PLEG): container finished" podID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" containerID="5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3" exitCode=0
Nov 02 22:26:17.247411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.246989    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"
Nov 02 22:26:17.247411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.247029    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" event={"ID":"21e473fb-93fc-4e27-9d9e-241ea3ea9b60","Type":"ContainerDied","Data":"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"}
Nov 02 22:26:17.247411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.247088    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp" event={"ID":"21e473fb-93fc-4e27-9d9e-241ea3ea9b60","Type":"ContainerDied","Data":"7f557f8d4b806c5f361fa73ba879d21bcfa8f70408cc3d34f9c6fa6896c7b1eb"}
Nov 02 22:26:17.247411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.247109    4929 scope.go:122] "RemoveContainer" containerID="5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"
Nov 02 22:26:17.257044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.257022    4929 scope.go:122] "RemoveContainer" containerID="5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"
Nov 02 22:26:17.257325 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:17.257303    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3\": container with ID starting with 5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3 not found: ID does not exist" containerID="5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"
Nov 02 22:26:17.257403 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.257333    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3"} err="failed to get container status \"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3\": rpc error: code = NotFound desc = could not find container \"5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3\": container with ID starting with 5d16cf7ad1da6d6fbca2fed6fcb512c60dc23008c27cf6b1780c4a030a9121d3 not found: ID does not exist"
Nov 02 22:26:17.267243 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.267220    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:26:17.269741 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.269723    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ac0910-56f74fd9dc-pnjwp"]
Nov 02 22:26:17.575429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.575340    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:17.655446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.655402    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main" containerID="cri-o://74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5" gracePeriod=45
Nov 02 22:26:17.924888 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:17.924864    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:26:18.075750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.075720    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.075933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.075774    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.075933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.075853    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.075933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.075925    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.076108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076020    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.076108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076069    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\" (UniqueName: \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\") pod \"2f3404cc-af7a-4aba-b97a-a62738a90624\" (UID: \"2f3404cc-af7a-4aba-b97a-a62738a90624\") "
Nov 02 22:26:18.076108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076074    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:18.076278 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076148    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:18.076328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076273    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:18.076600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076579    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.076699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076603    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.076699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.076617    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.078117 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.078092    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:18.078205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.078162    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:18.078348 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.078307    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz" pod "2f3404cc-af7a-4aba-b97a-a62738a90624" (UID: "2f3404cc-af7a-4aba-b97a-a62738a90624"). InnerVolumeSpecName "kube-api-access-bgqvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:18.177370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.177340    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqvz\" (UniqueName: \"kubernetes.io/projected/2f3404cc-af7a-4aba-b97a-a62738a90624-kube-api-access-bgqvz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.177370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.177369    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3404cc-af7a-4aba-b97a-a62738a90624-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.177660 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.177379    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3404cc-af7a-4aba-b97a-a62738a90624-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:18.265122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.265082    4929 generic.go:356] "Generic (PLEG): container finished" podID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerID="74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5" exitCode=0
Nov 02 22:26:18.265568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.265132    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"
Nov 02 22:26:18.265568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.265156    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" event={"ID":"2f3404cc-af7a-4aba-b97a-a62738a90624","Type":"ContainerDied","Data":"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"}
Nov 02 22:26:18.265568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.265201    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs" event={"ID":"2f3404cc-af7a-4aba-b97a-a62738a90624","Type":"ContainerDied","Data":"afed03889b5f8a4bc084304dc67943a7884cfc206011857ee6e679fd35f55f3c"}
Nov 02 22:26:18.265568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.265225    4929 scope.go:122] "RemoveContainer" containerID="74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"
Nov 02 22:26:18.277520 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.277501    4929 scope.go:122] "RemoveContainer" containerID="74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"
Nov 02 22:26:18.277862 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:18.277834    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5\": container with ID starting with 74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5 not found: ID does not exist" containerID="74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"
Nov 02 22:26:18.277953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.277870    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5"} err="failed to get container status \"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5\": rpc error: code = NotFound desc = could not find container \"74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5\": container with ID starting with 74f9d314651fcdef247df2e70861f8d4741e1426f8ba3f33cd623e4cdcd143c5 not found: ID does not exist"
Nov 02 22:26:18.288103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.288080    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:26:18.291578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.291555    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-719012/distinct-w3-719012-kserve-7f856fd7b8-7pqbs"]
Nov 02 22:26:18.803660 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.803627    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" path="/var/lib/kubelet/pods/21e473fb-93fc-4e27-9d9e-241ea3ea9b60/volumes"
Nov 02 22:26:18.804095 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:18.804081    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" path="/var/lib/kubelet/pods/2f3404cc-af7a-4aba-b97a-a62738a90624/volumes"
Nov 02 22:26:20.314469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:20.314434    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:26:20.314934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:20.314762    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy" containerID="cri-o://0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564" gracePeriod=30
Nov 02 22:26:20.645763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:20.645709    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing" containerID="cri-o://25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28" gracePeriod=25
Nov 02 22:26:23.527434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.527397    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:26:23.527955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.527708    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerName="maas-api" containerID="cri-o://8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f" gracePeriod=30
Nov 02 22:26:23.594209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.594167    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": dial tcp 10.133.0.47:8443: connect: connection refused"
Nov 02 22:26:23.850042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.850013    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:26:23.850830 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.850811    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\") pod \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") "
Nov 02 22:26:23.850948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.850849    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\") pod \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") "
Nov 02 22:26:23.850948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.850898    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\") pod \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") "
Nov 02 22:26:23.851076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.851015    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\" (UniqueName: \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\") pod \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\" (UID: \"0eb363f6-1190-4f04-8e86-1695bd76d7ee\") "
Nov 02 22:26:23.851223 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.851201    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca" pod "0eb363f6-1190-4f04-8e86-1695bd76d7ee" (UID: "0eb363f6-1190-4f04-8e86-1695bd76d7ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:23.851483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.851457    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb363f6-1190-4f04-8e86-1695bd76d7ee-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:23.853195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.853173    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs" pod "0eb363f6-1190-4f04-8e86-1695bd76d7ee" (UID: "0eb363f6-1190-4f04-8e86-1695bd76d7ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:23.853399 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.853383    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls" pod "0eb363f6-1190-4f04-8e86-1695bd76d7ee" (UID: "0eb363f6-1190-4f04-8e86-1695bd76d7ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:23.853442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.853381    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz" pod "0eb363f6-1190-4f04-8e86-1695bd76d7ee" (UID: "0eb363f6-1190-4f04-8e86-1695bd76d7ee"). InnerVolumeSpecName "kube-api-access-d7dsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:23.952564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.952514    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:23.952564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.952564    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb363f6-1190-4f04-8e86-1695bd76d7ee-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:23.952772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:23.952575    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dsz\" (UniqueName: \"kubernetes.io/projected/0eb363f6-1190-4f04-8e86-1695bd76d7ee-kube-api-access-d7dsz\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:24.310219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.310184    4929 generic.go:356] "Generic (PLEG): container finished" podID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerID="8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f" exitCode=0
Nov 02 22:26:24.310415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.310232    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"
Nov 02 22:26:24.310415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.310249    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" event={"ID":"0eb363f6-1190-4f04-8e86-1695bd76d7ee","Type":"ContainerDied","Data":"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"}
Nov 02 22:26:24.310415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.310291    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g" event={"ID":"0eb363f6-1190-4f04-8e86-1695bd76d7ee","Type":"ContainerDied","Data":"b635116e1bdc169449748b3671089d5bdbe164cded331b5c76c5a6b7a2c7b988"}
Nov 02 22:26:24.310415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.310312    4929 scope.go:122] "RemoveContainer" containerID="8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"
Nov 02 22:26:24.320515 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.320481    4929 scope.go:122] "RemoveContainer" containerID="8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"
Nov 02 22:26:24.320839 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:24.320805    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f\": container with ID starting with 8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f not found: ID does not exist" containerID="8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"
Nov 02 22:26:24.320895 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.320841    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f"} err="failed to get container status \"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f\": rpc error: code = NotFound desc = could not find container \"8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f\": container with ID starting with 8c880f2cac9f4156dda773e82c36302e7b0a4cb10e500d7a273dc354747a1c6f not found: ID does not exist"
Nov 02 22:26:24.332618 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.332595    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:26:24.335818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.335797    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-719012-6d56bd9cd8-9r29g"]
Nov 02 22:26:24.804563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:24.804513    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" path="/var/lib/kubelet/pods/0eb363f6-1190-4f04-8e86-1695bd76d7ee/volumes"
Nov 02 22:26:25.318989 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.318948    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" containerID="cri-o://62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73" gracePeriod=45
Nov 02 22:26:25.329357 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:25.329317    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode691aebd_6ff9_4836_8e7c_0f74f7bef077.slice/crio-62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 22:26:25.329357 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:25.329370    4929 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode691aebd_6ff9_4836_8e7c_0f74f7bef077.slice/crio-62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 22:26:25.336506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.336442    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main" containerID="cri-o://3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa" gracePeriod=45
Nov 02 22:26:25.617156 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.617124    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:26:25.662573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.661460    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:26:25.681115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.681081    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:26:25.775179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775142    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775207    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\" (UniqueName: \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.775374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775249    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.775374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775281    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775374 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775310    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775425    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775502    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\" (UniqueName: \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775548    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775580    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775621    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.775684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775659    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775653    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775687    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775769    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775811    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775849    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\") pod \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\" (UID: \"e691aebd-6ff9-4836-8e7c-0f74f7bef077\") "
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775856    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.775929    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\") pod \"43989402-9353-4abd-bf15-425b0381eaba\" (UID: \"43989402-9353-4abd-bf15-425b0381eaba\") "
Nov 02 22:26:25.776053 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776000    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776193    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776414    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776488    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776513    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776570    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776589    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776599    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.776621 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776603    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43989402-9353-4abd-bf15-425b0381eaba-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.776958 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.776803    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.777926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.777900    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:25.778042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.777976    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.778148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.778125    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.778264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.778245    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "kube-api-access-hcr7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:25.778327 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.778300    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts" pod "e691aebd-6ff9-4836-8e7c-0f74f7bef077" (UID: "e691aebd-6ff9-4836-8e7c-0f74f7bef077"). InnerVolumeSpecName "kube-api-access-7fgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:25.778702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.778676    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:25.779010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.778993    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo" pod "43989402-9353-4abd-bf15-425b0381eaba" (UID: "43989402-9353-4abd-bf15-425b0381eaba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:25.877703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877663    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\" (UniqueName: \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877743    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\" (UniqueName: \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877797    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877851    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877906    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.877976    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\") pod \"43a8b213-8821-404a-8f1b-a0cbbe32793b\" (UID: \"43a8b213-8821-404a-8f1b-a0cbbe32793b\") "
Nov 02 22:26:25.878220 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878137    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.878585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878266    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.878585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878419    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878595    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878617    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fgts\" (UniqueName: \"kubernetes.io/projected/e691aebd-6ff9-4836-8e7c-0f74f7bef077-kube-api-access-7fgts\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878635    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878650    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878665    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878682    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcr7p\" (UniqueName: \"kubernetes.io/projected/43989402-9353-4abd-bf15-425b0381eaba-kube-api-access-hcr7p\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878697    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878711    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878726    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43989402-9353-4abd-bf15-425b0381eaba-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878741    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43989402-9353-4abd-bf15-425b0381eaba-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878755    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e691aebd-6ff9-4836-8e7c-0f74f7bef077-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.878770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878773    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e691aebd-6ff9-4836-8e7c-0f74f7bef077-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.879396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.878786    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.880252 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.880229    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "kube-api-access-jhmcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:25.880429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.880405    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:25.880429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.880418    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm" pod "43a8b213-8821-404a-8f1b-a0cbbe32793b" (UID: "43a8b213-8821-404a-8f1b-a0cbbe32793b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:25.979834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.979791    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a8b213-8821-404a-8f1b-a0cbbe32793b-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.979834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.979828    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmcr\" (UniqueName: \"kubernetes.io/projected/43a8b213-8821-404a-8f1b-a0cbbe32793b-kube-api-access-jhmcr\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:25.980054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:25.979843    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a8b213-8821-404a-8f1b-a0cbbe32793b-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:26.324274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.324189    4929 generic.go:356] "Generic (PLEG): container finished" podID="43989402-9353-4abd-bf15-425b0381eaba" containerID="0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564" exitCode=0
Nov 02 22:26:26.324274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.324236    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"
Nov 02 22:26:26.324516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.324274    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerDied","Data":"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"}
Nov 02 22:26:26.324516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.324330    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2" event={"ID":"43989402-9353-4abd-bf15-425b0381eaba","Type":"ContainerDied","Data":"dd2fac835408c4233277ddf04cdee332c321eb675a77bcd61421713f5e4ffeeb"}
Nov 02 22:26:26.324516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.324363    4929 scope.go:122] "RemoveContainer" containerID="0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"
Nov 02 22:26:26.325827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.325636    4929 generic.go:356] "Generic (PLEG): container finished" podID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerID="62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73" exitCode=0
Nov 02 22:26:26.325827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.325684    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"
Nov 02 22:26:26.325827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.325714    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" event={"ID":"e691aebd-6ff9-4836-8e7c-0f74f7bef077","Type":"ContainerDied","Data":"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"}
Nov 02 22:26:26.325827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.325757    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" event={"ID":"e691aebd-6ff9-4836-8e7c-0f74f7bef077","Type":"ContainerDied","Data":"8d5dff0d9185a44688e99a288b4abf8c6617caed52b7d94e54d9cbbd136c550a"}
Nov 02 22:26:26.327367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.327258    4929 generic.go:356] "Generic (PLEG): container finished" podID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerID="3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa" exitCode=0
Nov 02 22:26:26.327367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.327302    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" event={"ID":"43a8b213-8821-404a-8f1b-a0cbbe32793b","Type":"ContainerDied","Data":"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"}
Nov 02 22:26:26.327367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.327316    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"
Nov 02 22:26:26.327367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.327334    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr" event={"ID":"43a8b213-8821-404a-8f1b-a0cbbe32793b","Type":"ContainerDied","Data":"cccb33285d884e95e21c65632886a1ca7b9bc4708ace168d3c59aad9bd8b9d2c"}
Nov 02 22:26:26.372088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.372062    4929 scope.go:122] "RemoveContainer" containerID="8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"
Nov 02 22:26:26.381410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.381388    4929 scope.go:122] "RemoveContainer" containerID="0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"
Nov 02 22:26:26.381750 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:26.381727    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564\": container with ID starting with 0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564 not found: ID does not exist" containerID="0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"
Nov 02 22:26:26.381823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.381758    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564"} err="failed to get container status \"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564\": rpc error: code = NotFound desc = could not find container \"0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564\": container with ID starting with 0fe4abb8a2c38f912d53b9fbac95c0857700914c57ca658b3a11053031cb0564 not found: ID does not exist"
Nov 02 22:26:26.381823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.381777    4929 scope.go:122] "RemoveContainer" containerID="8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"
Nov 02 22:26:26.382058 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:26.382044    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d\": container with ID starting with 8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d not found: ID does not exist" containerID="8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"
Nov 02 22:26:26.382116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.382060    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d"} err="failed to get container status \"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d\": rpc error: code = NotFound desc = could not find container \"8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d\": container with ID starting with 8b23d21f5bd9298e0ff8ed9139effdc5f9196c720f573c3a2ea9d7e915a22f8d not found: ID does not exist"
Nov 02 22:26:26.382116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.382071    4929 scope.go:122] "RemoveContainer" containerID="62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"
Nov 02 22:26:26.386446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.386415    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:26:26.389508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.389487    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/premium-w0-310bb3-kserve-6886cf5989-b9xjr"]
Nov 02 22:26:26.392866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.392849    4929 scope.go:122] "RemoveContainer" containerID="62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"
Nov 02 22:26:26.393192 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:26.393173    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73\": container with ID starting with 62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73 not found: ID does not exist" containerID="62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"
Nov 02 22:26:26.393257 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.393200    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73"} err="failed to get container status \"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73\": rpc error: code = NotFound desc = could not find container \"62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73\": container with ID starting with 62fa5b5429d8c4448377582f8bc5b926450e5df7dc5468e1719dc1a781cfab73 not found: ID does not exist"
Nov 02 22:26:26.393257 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.393219    4929 scope.go:122] "RemoveContainer" containerID="3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"
Nov 02 22:26:26.400323 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.400300    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:26:26.402514 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.402489    4929 scope.go:122] "RemoveContainer" containerID="3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"
Nov 02 22:26:26.402890 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:26.402867    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa\": container with ID starting with 3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa not found: ID does not exist" containerID="3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"
Nov 02 22:26:26.402987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.402898    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa"} err="failed to get container status \"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa\": rpc error: code = NotFound desc = could not find container \"3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa\": container with ID starting with 3482d773c86f68fd7aed1542637b6d1dec45581e715d29c00323c65a7e30ecaa not found: ID does not exist"
Nov 02 22:26:26.404785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.404763    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ac0910-openshift-default-774cb58f9c-gk4w2"]
Nov 02 22:26:26.414076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.414053    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:26:26.420009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.419989    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4"]
Nov 02 22:26:26.628590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.628550    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-2-w0-310bb3-kserve-795d6c7bbc-sw9f4" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:26:26.804805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.804774    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43989402-9353-4abd-bf15-425b0381eaba" path="/var/lib/kubelet/pods/43989402-9353-4abd-bf15-425b0381eaba/volumes"
Nov 02 22:26:26.805264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.805249    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" path="/var/lib/kubelet/pods/43a8b213-8821-404a-8f1b-a0cbbe32793b/volumes"
Nov 02 22:26:26.805688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:26.805674    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" path="/var/lib/kubelet/pods/e691aebd-6ff9-4836-8e7c-0f74f7bef077/volumes"
Nov 02 22:26:27.497103 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:27.497068    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:26:27.575372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:27.575326    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:28.593298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:28.593257    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:30.848578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:30.848489    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:26:30.853571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:30.850325    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy" containerID="cri-o://bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e" gracePeriod=30
Nov 02 22:26:30.858990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:30.858963    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:26:30.859208 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:30.859170    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" containerID="cri-o://0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59" gracePeriod=30
Nov 02 22:26:31.018338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.016632    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:26:31.018338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.016863    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy" containerID="cri-o://0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea" gracePeriod=30
Nov 02 22:26:31.323800 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.323761    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main" containerID="cri-o://e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9" gracePeriod=45
Nov 02 22:26:31.349448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.349414    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main" containerID="cri-o://ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0" gracePeriod=45
Nov 02 22:26:31.684308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.684282    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:26:31.686977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.686946    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:26:31.761029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.760996    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\" (UniqueName: \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761049    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761072    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761105    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761196 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761142    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761231    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761296    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761342    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.761425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761355    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761389    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761436    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\" (UniqueName: \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761474    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\" (UniqueName: \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\") pod \"fae75d80-8497-4240-80d7-7b55d7b97b37\" (UID: \"fae75d80-8497-4240-80d7-7b55d7b97b37\") "
Nov 02 22:26:31.761809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761590    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\") pod \"1b413e10-d99a-487b-bdf5-371604909fe0\" (UID: \"1b413e10-d99a-487b-bdf5-371604909fe0\") "
Nov 02 22:26:31.761809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761672    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.762090 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.761870    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.762157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762085    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762281    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762710    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762734    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762747    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762761    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762774    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.762886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.762848    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.764162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.764141    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.764758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.764732    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "kube-api-access-p86wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:31.765062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.765039    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:31.765144 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.765073    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "kube-api-access-lskmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:31.766041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.766015    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm" pod "fae75d80-8497-4240-80d7-7b55d7b97b37" (UID: "fae75d80-8497-4240-80d7-7b55d7b97b37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:31.766410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.766385    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs" pod "1b413e10-d99a-487b-bdf5-371604909fe0" (UID: "1b413e10-d99a-487b-bdf5-371604909fe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:31.850589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.850527    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863899    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863941    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b413e10-d99a-487b-bdf5-371604909fe0-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863955    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae75d80-8497-4240-80d7-7b55d7b97b37-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863969    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86wf\" (UniqueName: \"kubernetes.io/projected/fae75d80-8497-4240-80d7-7b55d7b97b37-kube-api-access-p86wf\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863983    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lskmc\" (UniqueName: \"kubernetes.io/projected/1b413e10-d99a-487b-bdf5-371604909fe0-kube-api-access-lskmc\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.863998    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae75d80-8497-4240-80d7-7b55d7b97b37-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:31.864038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:31.864012    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b413e10-d99a-487b-bdf5-371604909fe0-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:32.319897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.319861    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:26:32.321569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.321514    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" podUID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" containerName="maas-api" containerID="cri-o://fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a" gracePeriod=30
Nov 02 22:26:32.367940 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.367904    4929 generic.go:356] "Generic (PLEG): container finished" podID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerID="ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0" exitCode=0
Nov 02 22:26:32.368091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.367950    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"
Nov 02 22:26:32.368091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.367984    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" event={"ID":"fae75d80-8497-4240-80d7-7b55d7b97b37","Type":"ContainerDied","Data":"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"}
Nov 02 22:26:32.368091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.368031    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh" event={"ID":"fae75d80-8497-4240-80d7-7b55d7b97b37","Type":"ContainerDied","Data":"ca077dfdb94c7f2f1f68f6785b558cf1d0bf02221d309f971d391f971d71c2a2"}
Nov 02 22:26:32.368091 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.368054    4929 scope.go:122] "RemoveContainer" containerID="ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"
Nov 02 22:26:32.369638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.369594    4929 generic.go:356] "Generic (PLEG): container finished" podID="1b413e10-d99a-487b-bdf5-371604909fe0" containerID="e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9" exitCode=0
Nov 02 22:26:32.369638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.369637    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"
Nov 02 22:26:32.369820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.369677    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" event={"ID":"1b413e10-d99a-487b-bdf5-371604909fe0","Type":"ContainerDied","Data":"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"}
Nov 02 22:26:32.369820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.369705    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r" event={"ID":"1b413e10-d99a-487b-bdf5-371604909fe0","Type":"ContainerDied","Data":"2bf72c4738b0e02ee1a03f3b9ae4206f23c6607a6a132074ef08b35c61e56fad"}
Nov 02 22:26:32.400975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.400945    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:26:32.408193 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.408169    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/simulator-w2-2532af-kserve-7f98c68b76-8nxxh"]
Nov 02 22:26:32.421709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.421677    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:26:32.426122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.426097    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-2-w2-2532af-kserve-6b5bb58c59-4sj8r"]
Nov 02 22:26:32.437699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.437676    4929 scope.go:122] "RemoveContainer" containerID="ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"
Nov 02 22:26:32.438063 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:32.438042    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0\": container with ID starting with ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0 not found: ID does not exist" containerID="ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"
Nov 02 22:26:32.438147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.438073    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0"} err="failed to get container status \"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0\": rpc error: code = NotFound desc = could not find container \"ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0\": container with ID starting with ad761a09d96a9426e388a1635c9aec6c51c6c3c98bc283e9d94b69509b7707d0 not found: ID does not exist"
Nov 02 22:26:32.438147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.438091    4929 scope.go:122] "RemoveContainer" containerID="e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"
Nov 02 22:26:32.446384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.446357    4929 scope.go:122] "RemoveContainer" containerID="e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"
Nov 02 22:26:32.446705 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:32.446682    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9\": container with ID starting with e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9 not found: ID does not exist" containerID="e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"
Nov 02 22:26:32.446807 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.446710    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9"} err="failed to get container status \"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9\": rpc error: code = NotFound desc = could not find container \"e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9\": container with ID starting with e50e33ada48bdaa10bcdd6d82872dc9788666f227a5a5c00ca6d8a0001ae5be9 not found: ID does not exist"
Nov 02 22:26:32.662993 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.662969    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:26:32.779097 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.779063    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\") pod \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") "
Nov 02 22:26:32.779284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.779111    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\") pod \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") "
Nov 02 22:26:32.779284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.779144    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\") pod \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") "
Nov 02 22:26:32.779404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.779342    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\" (UniqueName: \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\") pod \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\" (UID: \"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c\") "
Nov 02 22:26:32.780146 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.780078    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca" pod "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" (UID: "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:32.781923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.781304    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:32.786873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.782241    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs" pod "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" (UID: "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:32.786873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.782301    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr" pod "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" (UID: "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c"). InnerVolumeSpecName "kube-api-access-57fdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:32.786873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.782484    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls" pod "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" (UID: "4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:32.804757 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.804728    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" path="/var/lib/kubelet/pods/1b413e10-d99a-487b-bdf5-371604909fe0/volumes"
Nov 02 22:26:32.805181 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.805167    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" path="/var/lib/kubelet/pods/fae75d80-8497-4240-80d7-7b55d7b97b37/volumes"
Nov 02 22:26:32.882443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.882404    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:32.882443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.882436    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:32.882443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:32.882449    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fdr\" (UniqueName: \"kubernetes.io/projected/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c-kube-api-access-57fdr\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:33.007694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.007608    4929 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:26:33.007694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.007667    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:26:33.314582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.314556    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:26:33.377138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.377106    4929 generic.go:356] "Generic (PLEG): container finished" podID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerID="4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d" exitCode=0
Nov 02 22:26:33.377320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.377170    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"
Nov 02 22:26:33.377320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.377197    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" event={"ID":"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062","Type":"ContainerDied","Data":"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"}
Nov 02 22:26:33.377320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.377266    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq" event={"ID":"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062","Type":"ContainerDied","Data":"6c5b0933d4c34d494286766fad6b9ebd91e8263d35f4cdf7c1198a49ec0e9a30"}
Nov 02 22:26:33.377320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.377299    4929 scope.go:122] "RemoveContainer" containerID="4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"
Nov 02 22:26:33.379806 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.379780    4929 generic.go:356] "Generic (PLEG): container finished" podID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" containerID="fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a" exitCode=0
Nov 02 22:26:33.379951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.379836    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" event={"ID":"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c","Type":"ContainerDied","Data":"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"}
Nov 02 22:26:33.379951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.379904    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh" event={"ID":"4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c","Type":"ContainerDied","Data":"9968299f91dc133eb0d6b62eb17b02a6b275bbbce02e27a161c753e516218817"}
Nov 02 22:26:33.379951 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.379844    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"
Nov 02 22:26:33.387917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.387889    4929 scope.go:122] "RemoveContainer" containerID="4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"
Nov 02 22:26:33.388222 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:33.388186    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d\": container with ID starting with 4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d not found: ID does not exist" containerID="4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"
Nov 02 22:26:33.388310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.388227    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d"} err="failed to get container status \"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d\": rpc error: code = NotFound desc = could not find container \"4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d\": container with ID starting with 4533071fddf76d7e02b4df5237fa24065f4daff0b9c5573188ede7bdd736765d not found: ID does not exist"
Nov 02 22:26:33.388310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.388249    4929 scope.go:122] "RemoveContainer" containerID="fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"
Nov 02 22:26:33.388424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.388352    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\") pod \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") "
Nov 02 22:26:33.388424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.388400    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\" (UniqueName: \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\") pod \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\" (UID: \"5776cdc7-9c34-4d5a-b6ab-6cfc19a55062\") "
Nov 02 22:26:33.388767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.388744    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume" pod "5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" (UID: "5776cdc7-9c34-4d5a-b6ab-6cfc19a55062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:33.389225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.389204    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:33.391151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.391126    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7" pod "5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" (UID: "5776cdc7-9c34-4d5a-b6ab-6cfc19a55062"). InnerVolumeSpecName "kube-api-access-9sgv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:33.399659 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.399637    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:26:33.407675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.407650    4929 scope.go:122] "RemoveContainer" containerID="fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"
Nov 02 22:26:33.407760 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.407717    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-310bb3-8fb487db4-gskbh"]
Nov 02 22:26:33.407972 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:33.407954    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a\": container with ID starting with fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a not found: ID does not exist" containerID="fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"
Nov 02 22:26:33.408016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.407979    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a"} err="failed to get container status \"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a\": rpc error: code = NotFound desc = could not find container \"fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a\": container with ID starting with fc4fcb4e099c7fed25de112419f9e4f2cb95355669daa16ec8bbad4c46dbfd4a not found: ID does not exist"
Nov 02 22:26:33.490447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.490415    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sgv7\" (UniqueName: \"kubernetes.io/projected/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062-kube-api-access-9sgv7\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:33.704394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.704357    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:26:33.709281 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:33.709245    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-719012-79f4cbb575-dqbnq"]
Nov 02 22:26:34.803331 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:34.803290    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" path="/var/lib/kubelet/pods/4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c/volumes"
Nov 02 22:26:34.803743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:34.803680    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" path="/var/lib/kubelet/pods/5776cdc7-9c34-4d5a-b6ab-6cfc19a55062/volumes"
Nov 02 22:26:36.193203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.193160    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:26:36.196812 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.196787    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:26:36.287471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.287446    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:26:36.323960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.323920    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.323960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.323955    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.323992    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324021    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324055    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324119    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324146    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.324224 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324190    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324249    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324232    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324289    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324297    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324363    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324407    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324460    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324478    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324511    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.324594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324590    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324600    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324646    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324680    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324758    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324838    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324851    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324891    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324930    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324987    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.325103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325060    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325112    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325150    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\") pod \"747b4185-405d-48f1-913c-fafe99cd3d82\" (UID: \"747b4185-405d-48f1-913c-fafe99cd3d82\") "
Nov 02 22:26:36.325685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325217    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\") pod \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\" (UID: \"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e\") "
Nov 02 22:26:36.325685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325264    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\") pod \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\" (UID: \"b1ff629e-baab-46c7-a7d5-d04a3f0e414d\") "
Nov 02 22:26:36.325685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.324925    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.325966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325854    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747b4185-405d-48f1-913c-fafe99cd3d82-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.325966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325921    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.325966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325938    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.325966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325954    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.326178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325968    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.326178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.326043    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:36.326178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.326172    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.326347 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.326188    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.326679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.326649    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.326795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325338    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.326795 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.325396    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.326919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.326856    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.327162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.327139    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.327424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.327389    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.329915    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "kube-api-access-h8wt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330003    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330033    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330049    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330061    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330106    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "kube-api-access-hrj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.330158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330131    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token" pod "b1ff629e-baab-46c7-a7d5-d04a3f0e414d" (UID: "b1ff629e-baab-46c7-a7d5-d04a3f0e414d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.330683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330202    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:36.330683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330509    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:36.330952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.330932    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token" pod "747b4185-405d-48f1-913c-fafe99cd3d82" (UID: "747b4185-405d-48f1-913c-fafe99cd3d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.331587 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.331563    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.331881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.331858    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww" pod "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" (UID: "772136e8-cdeb-4b26-8d15-e7cf8a1bb28e"). InnerVolumeSpecName "kube-api-access-t59ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:36.401880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.401794    4929 generic.go:356] "Generic (PLEG): container finished" podID="747b4185-405d-48f1-913c-fafe99cd3d82" containerID="bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e" exitCode=0
Nov 02 22:26:36.401880 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.401847    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"
Nov 02 22:26:36.402106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.401880    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" event={"ID":"747b4185-405d-48f1-913c-fafe99cd3d82","Type":"ContainerDied","Data":"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"}
Nov 02 22:26:36.402106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.401955    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl" event={"ID":"747b4185-405d-48f1-913c-fafe99cd3d82","Type":"ContainerDied","Data":"4f062f1f9fad377c79c4eb38e0fb2b130cbd1b8018f99ea099f0a9dfaf646c64"}
Nov 02 22:26:36.402106 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.401984    4929 scope.go:122] "RemoveContainer" containerID="bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"
Nov 02 22:26:36.403392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.403372    4929 generic.go:356] "Generic (PLEG): container finished" podID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerID="0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea" exitCode=0
Nov 02 22:26:36.403519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.403424    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"
Nov 02 22:26:36.403519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.403460    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" event={"ID":"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e","Type":"ContainerDied","Data":"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"}
Nov 02 22:26:36.403519 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.403512    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf" event={"ID":"772136e8-cdeb-4b26-8d15-e7cf8a1bb28e","Type":"ContainerDied","Data":"20a44b5ac5694cead6751c39a3445b3bcbf6ffde963a83d524bb5d54ee53cb8d"}
Nov 02 22:26:36.406704 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.406683    4929 generic.go:356] "Generic (PLEG): container finished" podID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerID="0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59" exitCode=0
Nov 02 22:26:36.406808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.406727    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" event={"ID":"b1ff629e-baab-46c7-a7d5-d04a3f0e414d","Type":"ContainerDied","Data":"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"}
Nov 02 22:26:36.406808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.406757    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk" event={"ID":"b1ff629e-baab-46c7-a7d5-d04a3f0e414d","Type":"ContainerDied","Data":"eb46870b4e30a3240d2f2bf9b92c4bc647152a7887e14929c8a103c14c2e0af0"}
Nov 02 22:26:36.406933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.406901    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"
Nov 02 22:26:36.414089 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.413848    4929 scope.go:122] "RemoveContainer" containerID="bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"
Nov 02 22:26:36.414192 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:36.414151    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e\": container with ID starting with bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e not found: ID does not exist" containerID="bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"
Nov 02 22:26:36.414264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.414183    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e"} err="failed to get container status \"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e\": rpc error: code = NotFound desc = could not find container \"bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e\": container with ID starting with bb48ae93e911fb07d58aaead27a147655fca6fdf9939bdb78e342deb21d8cc7e not found: ID does not exist"
Nov 02 22:26:36.414264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.414204    4929 scope.go:122] "RemoveContainer" containerID="0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"
Nov 02 22:26:36.426271 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.426247    4929 scope.go:122] "RemoveContainer" containerID="0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"
Nov 02 22:26:36.426654 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:36.426628    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea\": container with ID starting with 0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea not found: ID does not exist" containerID="0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"
Nov 02 22:26:36.426743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.426667    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea"} err="failed to get container status \"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea\": rpc error: code = NotFound desc = could not find container \"0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea\": container with ID starting with 0a6b35dd89781cd12d07f60b4c868e3000af5c84b437d13f54580cddb13f3aea not found: ID does not exist"
Nov 02 22:26:36.426743 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.426689    4929 scope.go:122] "RemoveContainer" containerID="0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"
Nov 02 22:26:36.427028 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427005    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427031    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427047    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427061    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747b4185-405d-48f1-913c-fafe99cd3d82-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427077    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427090    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427103    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427115    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427128    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427143    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427157    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427166    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427175    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wt9\" (UniqueName: \"kubernetes.io/projected/747b4185-405d-48f1-913c-fafe99cd3d82-kube-api-access-h8wt9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427186    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427202    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427213    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrj5g\" (UniqueName: \"kubernetes.io/projected/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-kube-api-access-hrj5g\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427226    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff629e-baab-46c7-a7d5-d04a3f0e414d-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427239    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427254    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747b4185-405d-48f1-913c-fafe99cd3d82-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427265    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427280    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59ww\" (UniqueName: \"kubernetes.io/projected/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-kube-api-access-t59ww\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.427470 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.427291    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:36.432834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.432809    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.438357    4929 scope.go:122] "RemoveContainer" containerID="0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:36.438719    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59\": container with ID starting with 0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59 not found: ID does not exist" containerID="0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.438746    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59"} err="failed to get container status \"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59\": rpc error: code = NotFound desc = could not find container \"0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59\": container with ID starting with 0f5021ca87c25265457795ae13e5e367dbc054d3ea5727eb54ea269e23bfee59 not found: ID does not exist"
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.441868    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-sjrfl"]
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.451261    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:26:36.458528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.458337    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-g66jf"]
Nov 02 22:26:36.469085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.469061    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:26:36.472682 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.472662    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-w7wvk"]
Nov 02 22:26:36.803850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.803771    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" path="/var/lib/kubelet/pods/747b4185-405d-48f1-913c-fafe99cd3d82/volumes"
Nov 02 22:26:36.804342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.804320    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" path="/var/lib/kubelet/pods/772136e8-cdeb-4b26-8d15-e7cf8a1bb28e/volumes"
Nov 02 22:26:36.804848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:36.804829    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" path="/var/lib/kubelet/pods/b1ff629e-baab-46c7-a7d5-d04a3f0e414d/volumes"
Nov 02 22:26:38.593330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:38.593287    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:26:40.910443 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:40.910374    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:26:40.982138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:40.982105    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\" (UniqueName: \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\") pod \"5d35874e-a66a-4509-8705-6f359c931155\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") "
Nov 02 22:26:40.982345 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:40.982160    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\") pod \"5d35874e-a66a-4509-8705-6f359c931155\" (UID: \"5d35874e-a66a-4509-8705-6f359c931155\") "
Nov 02 22:26:40.982599 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:40.982567    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume" pod "5d35874e-a66a-4509-8705-6f359c931155" (UID: "5d35874e-a66a-4509-8705-6f359c931155"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:40.984262 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:40.984237    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh" pod "5d35874e-a66a-4509-8705-6f359c931155" (UID: "5d35874e-a66a-4509-8705-6f359c931155"). InnerVolumeSpecName "kube-api-access-wxhnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:41.083309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.083268    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35874e-a66a-4509-8705-6f359c931155-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:41.083309 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.083307    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhnh\" (UniqueName: \"kubernetes.io/projected/5d35874e-a66a-4509-8705-6f359c931155-kube-api-access-wxhnh\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:41.288350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.288260    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:26:41.288508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.288471    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy" containerID="cri-o://a37407eb222607fa1987168e16e933ef6138edf389515e4f5b145e55bd24db8d" gracePeriod=30
Nov 02 22:26:41.439702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.439668    4929 generic.go:356] "Generic (PLEG): container finished" podID="5d35874e-a66a-4509-8705-6f359c931155" containerID="25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28" exitCode=0
Nov 02 22:26:41.439919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.439724    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"
Nov 02 22:26:41.439919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.439759    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" event={"ID":"5d35874e-a66a-4509-8705-6f359c931155","Type":"ContainerDied","Data":"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"}
Nov 02 22:26:41.439919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.439817    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86" event={"ID":"5d35874e-a66a-4509-8705-6f359c931155","Type":"ContainerDied","Data":"e0110da6a48b69f8f159c4ac88997d78634e5758ea13658d883b6650ef66ea79"}
Nov 02 22:26:41.439919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.439845    4929 scope.go:122] "RemoveContainer" containerID="25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"
Nov 02 22:26:41.449873 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.449845    4929 scope.go:122] "RemoveContainer" containerID="25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"
Nov 02 22:26:41.450164 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:41.450141    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28\": container with ID starting with 25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28 not found: ID does not exist" containerID="25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"
Nov 02 22:26:41.450258 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.450171    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28"} err="failed to get container status \"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28\": rpc error: code = NotFound desc = could not find container \"25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28\": container with ID starting with 25ce092a04c460172d3b89915e9031f25630978298f2c38051110fe5dce80f28 not found: ID does not exist"
Nov 02 22:26:41.462026 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.461995    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:26:41.463955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:41.463872    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-310bb3-6c89787cbf-6vn86"]
Nov 02 22:26:42.318825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:42.318789    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:26:42.319362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:42.319028    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy" containerID="cri-o://021c753bc0535e987794a02473be995813d5650863ab568f00602268f4824a06" gracePeriod=30
Nov 02 22:26:42.323415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:42.323386    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:26:42.323653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:42.323609    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy" containerID="cri-o://25080e7f7b4fefafc8629ab803963c2d803c5a6e25670b766f6bddfdb2fdf16e" gracePeriod=30
Nov 02 22:26:42.810666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:42.810632    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d35874e-a66a-4509-8705-6f359c931155" path="/var/lib/kubelet/pods/5d35874e-a66a-4509-8705-6f359c931155/volumes"
Nov 02 22:26:46.474742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.474710    4929 generic.go:356] "Generic (PLEG): container finished" podID="99242638-804b-46e5-ac4e-cd57dfef944f" containerID="a37407eb222607fa1987168e16e933ef6138edf389515e4f5b145e55bd24db8d" exitCode=0
Nov 02 22:26:46.475122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.474797    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" event={"ID":"99242638-804b-46e5-ac4e-cd57dfef944f","Type":"ContainerDied","Data":"a37407eb222607fa1987168e16e933ef6138edf389515e4f5b145e55bd24db8d"}
Nov 02 22:26:46.561950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.561881    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:26:46.659186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659155    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659191    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659328    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659439    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659470    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659507    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659517    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659590    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659629    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.659670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659656    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.660187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659701    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\") pod \"99242638-804b-46e5-ac4e-cd57dfef944f\" (UID: \"99242638-804b-46e5-ac4e-cd57dfef944f\") "
Nov 02 22:26:46.660187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.659756    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:46.660187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.660029    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99242638-804b-46e5-ac4e-cd57dfef944f-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.660187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.660049    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.660187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.660065    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.660455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.660225    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:46.660455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.660437    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:46.662121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.662091    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:46.662255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.662237    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:46.662387 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.662364    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:46.664180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.664153    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc" pod "99242638-804b-46e5-ac4e-cd57dfef944f" (UID: "99242638-804b-46e5-ac4e-cd57dfef944f"). InnerVolumeSpecName "kube-api-access-zpkmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:46.761411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761376    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.761411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761406    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99242638-804b-46e5-ac4e-cd57dfef944f-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.761411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761417    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.761707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761426    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkmc\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-kube-api-access-zpkmc\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.761707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761437    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99242638-804b-46e5-ac4e-cd57dfef944f-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:46.761707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:46.761445    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99242638-804b-46e5-ac4e-cd57dfef944f-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.028310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.028279    4929 scope.go:122] "RemoveContainer" containerID="a37407eb222607fa1987168e16e933ef6138edf389515e4f5b145e55bd24db8d"
Nov 02 22:26:47.486056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.486022    4929 generic.go:356] "Generic (PLEG): container finished" podID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerID="25080e7f7b4fefafc8629ab803963c2d803c5a6e25670b766f6bddfdb2fdf16e" exitCode=0
Nov 02 22:26:47.486574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.486102    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerDied","Data":"25080e7f7b4fefafc8629ab803963c2d803c5a6e25670b766f6bddfdb2fdf16e"}
Nov 02 22:26:47.486574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.486175    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv" event={"ID":"99242638-804b-46e5-ac4e-cd57dfef944f","Type":"ContainerDied","Data":"a79f61614167130ec7e58a86ac462c9e4a53e32bc569b1771f0cf2c6409cc61c"}
Nov 02 22:26:47.486574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.486218    4929 scope.go:122] "RemoveContainer" containerID="04b9f3725e23b2149b465e6c19a22957de60a4cf563264d8c201341dc19cca5e"
Nov 02 22:26:47.488610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.488283    4929 generic.go:356] "Generic (PLEG): container finished" podID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerID="021c753bc0535e987794a02473be995813d5650863ab568f00602268f4824a06" exitCode=0
Nov 02 22:26:47.488610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.488437    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" event={"ID":"2568beb3-4483-41c4-8f42-8fba1a39ea2b","Type":"ContainerDied","Data":"021c753bc0535e987794a02473be995813d5650863ab568f00602268f4824a06"}
Nov 02 22:26:47.488610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.488458    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"
Nov 02 22:26:47.524864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.524839    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:26:47.536125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.536093    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-gshfv"]
Nov 02 22:26:47.608310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.608286    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:26:47.626863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.626804    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:26:47.772626 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772590    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772645    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772679    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772699    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772741    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772772    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.772820 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772812    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.773169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772846    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772873    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.772946    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773023    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.773169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773074    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.773459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773287    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.773459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773328    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.773459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773404    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:47.773459 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773437    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773515    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773587    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.773705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773645    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.773705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773682    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.773934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773711    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\") pod \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\" (UID: \"7288e662-5c7b-4c14-8379-f3bac4ff1c2c\") "
Nov 02 22:26:47.773934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773782    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.773934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.773836    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\") pod \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\" (UID: \"2568beb3-4483-41c4-8f42-8fba1a39ea2b\") "
Nov 02 22:26:47.774573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.774249    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.774573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.774277    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.774573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.774292    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.774573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.774306    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.774573 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.774319    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.775042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775011    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775242    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775422    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775424    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "kube-api-access-42dz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:47.775645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775618    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:47.775710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775618    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:47.775710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775625    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775630    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775718    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:47.775936 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.775916    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:47.776126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.776104    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:47.776300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.776286    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7" pod "2568beb3-4483-41c4-8f42-8fba1a39ea2b" (UID: "2568beb3-4483-41c4-8f42-8fba1a39ea2b"). InnerVolumeSpecName "kube-api-access-bv7t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:47.777100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.777085    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo" pod "7288e662-5c7b-4c14-8379-f3bac4ff1c2c" (UID: "7288e662-5c7b-4c14-8379-f3bac4ff1c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875159    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875188    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv7t7\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-kube-api-access-bv7t7\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875199    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875209    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875218    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875227    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875235    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875244    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875252    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875260    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2568beb3-4483-41c4-8f42-8fba1a39ea2b-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875259 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875268    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dz9\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-kube-api-access-42dz9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875278    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:47.875804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:47.875286    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7288e662-5c7b-4c14-8379-f3bac4ff1c2c-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:48.431710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.431678    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:26:48.461791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.461755    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:26:48.494846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.494814    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"
Nov 02 22:26:48.495380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.494853    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8" event={"ID":"2568beb3-4483-41c4-8f42-8fba1a39ea2b","Type":"ContainerDied","Data":"217ebc732b864b87e2b5815ba65d36b9c4a42c309bfc06a431057b143b2efc47"}
Nov 02 22:26:48.495380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.494919    4929 scope.go:122] "RemoveContainer" containerID="021c753bc0535e987794a02473be995813d5650863ab568f00602268f4824a06"
Nov 02 22:26:48.496169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.496144    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt" event={"ID":"7288e662-5c7b-4c14-8379-f3bac4ff1c2c","Type":"ContainerDied","Data":"f8f8f824c132efe8fc14960181d37942c98adb3eb2e854b9ddbda523f8f95fb0"}
Nov 02 22:26:48.496250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.496168    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"
Nov 02 22:26:48.506024 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.505938    4929 scope.go:122] "RemoveContainer" containerID="25080e7f7b4fefafc8629ab803963c2d803c5a6e25670b766f6bddfdb2fdf16e"
Nov 02 22:26:48.596457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.596422    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:26:48.614713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.614634    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-hgtqt"]
Nov 02 22:26:48.626908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.626877    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:26:48.630983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.630949    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nmgv8"]
Nov 02 22:26:48.804255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.804215    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" path="/var/lib/kubelet/pods/2568beb3-4483-41c4-8f42-8fba1a39ea2b/volumes"
Nov 02 22:26:48.804923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.804906    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" path="/var/lib/kubelet/pods/7288e662-5c7b-4c14-8379-f3bac4ff1c2c/volumes"
Nov 02 22:26:48.805349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:48.805336    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" path="/var/lib/kubelet/pods/99242638-804b-46e5-ac4e-cd57dfef944f/volumes"
Nov 02 22:26:51.656020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:26:51.655984    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:26:57.495925 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:26:57.495842    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:27:03.451653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.451615    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main" containerID="cri-o://5fc70c55992f2995b0477b29da6a35df3eae1269b42b74e09e6f3ba178e4b150" gracePeriod=45
Nov 02 22:27:03.481272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.481212    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main" containerID="cri-o://828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec" gracePeriod=45
Nov 02 22:27:03.572693 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.572658    4929 generic.go:356] "Generic (PLEG): container finished" podID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerID="5fc70c55992f2995b0477b29da6a35df3eae1269b42b74e09e6f3ba178e4b150" exitCode=0
Nov 02 22:27:03.572859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.572738    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" event={"ID":"1a10639a-ee4d-4dbe-a12a-a2573ceedd24","Type":"ContainerDied","Data":"5fc70c55992f2995b0477b29da6a35df3eae1269b42b74e09e6f3ba178e4b150"}
Nov 02 22:27:03.731172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.731142    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:27:03.760292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.760259    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:27:03.839792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.839749    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.839980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.839807    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\" (UniqueName: \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.839980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.839864    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.839980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.839893    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.839980 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.839948    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\" (UniqueName: \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.840229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840007    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.840229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840044    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\" (UniqueName: \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.840229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840103    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.840229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840151    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.840229 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840147    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.840524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840268    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.840524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840295    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\") pod \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\" (UID: \"1a10639a-ee4d-4dbe-a12a-a2573ceedd24\") "
Nov 02 22:27:03.840524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840333    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\") pod \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\" (UID: \"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b\") "
Nov 02 22:27:03.840524 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840377    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.840777 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840733    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.840877 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840855    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.840938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840882    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.840938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.840896    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.842068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.841407    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.842068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.841719    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.842068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.842024    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.842886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.842841    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "kube-api-access-lm57l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:03.842886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.842866    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:03.843190 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.843161    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:03.843279 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.843246    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm" pod "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" (UID: "76ec59c0-fd8f-4d96-b56d-5be8c9e1633b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.843702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.843683    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "kube-api-access-xmxlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:03.844157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.844135    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm" pod "1a10639a-ee4d-4dbe-a12a-a2573ceedd24" (UID: "1a10639a-ee4d-4dbe-a12a-a2573ceedd24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:03.941637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941595    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941632    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941642    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941651    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm57l\" (UniqueName: \"kubernetes.io/projected/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-kube-api-access-lm57l\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941661    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941672    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxlj\" (UniqueName: \"kubernetes.io/projected/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-kube-api-access-xmxlj\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941679    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941688    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:03.941892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:03.941696    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a10639a-ee4d-4dbe-a12a-a2573ceedd24-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:04.579041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.579008    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"
Nov 02 22:27:04.579553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.579045    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t" event={"ID":"1a10639a-ee4d-4dbe-a12a-a2573ceedd24","Type":"ContainerDied","Data":"8b4ee373362c85a61722866eb3ec0c9f407de3248840ecb55d9370cf5a740905"}
Nov 02 22:27:04.579553 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.579093    4929 scope.go:122] "RemoveContainer" containerID="5fc70c55992f2995b0477b29da6a35df3eae1269b42b74e09e6f3ba178e4b150"
Nov 02 22:27:04.580491 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.580461    4929 generic.go:356] "Generic (PLEG): container finished" podID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerID="828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec" exitCode=0
Nov 02 22:27:04.580610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.580508    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" event={"ID":"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b","Type":"ContainerDied","Data":"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"}
Nov 02 22:27:04.580610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.580509    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"
Nov 02 22:27:04.580610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.580557    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql" event={"ID":"76ec59c0-fd8f-4d96-b56d-5be8c9e1633b","Type":"ContainerDied","Data":"dd8b869926deeea2dce8df2f25b007c57ad01af7a189167a46f1a3567508c3ac"}
Nov 02 22:27:04.590138 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.590121    4929 scope.go:122] "RemoveContainer" containerID="828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"
Nov 02 22:27:04.598228 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.598209    4929 scope.go:122] "RemoveContainer" containerID="828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"
Nov 02 22:27:04.598581 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:27:04.598511    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec\": container with ID starting with 828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec not found: ID does not exist" containerID="828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"
Nov 02 22:27:04.598581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.598560    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec"} err="failed to get container status \"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec\": rpc error: code = NotFound desc = could not find container \"828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec\": container with ID starting with 828a03ae712074ad7df0ec346aafb64fcac39f59f3d9ca8981a1f2545606baec not found: ID does not exist"
Nov 02 22:27:04.601119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.601095    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:27:04.606418 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.606396    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-2-w5-361a44-kserve-7896b6d6bd-q596t"]
Nov 02 22:27:04.616022 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.615981    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:27:04.618164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.618143    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/distinct-w5-361a44-kserve-865769cb79-jtpql"]
Nov 02 22:27:04.802925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.802848    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" path="/var/lib/kubelet/pods/1a10639a-ee4d-4dbe-a12a-a2573ceedd24/volumes"
Nov 02 22:27:04.803288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:04.803275    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" path="/var/lib/kubelet/pods/76ec59c0-fd8f-4d96-b56d-5be8c9e1633b/volumes"
Nov 02 22:27:06.439350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:06.439316    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:27:09.369485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.369447    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:27:09.369971 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.369732    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" podUID="2a2fc86c-dde3-49c5-aa03-600f697954fd" containerName="maas-api" containerID="cri-o://7974a1565153e052385df152e8b689569333e24d5b4d1bd8d8da1d1ffaf8a6b2" gracePeriod=30
Nov 02 22:27:09.608341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.608311    4929 generic.go:356] "Generic (PLEG): container finished" podID="2a2fc86c-dde3-49c5-aa03-600f697954fd" containerID="7974a1565153e052385df152e8b689569333e24d5b4d1bd8d8da1d1ffaf8a6b2" exitCode=0
Nov 02 22:27:09.608494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.608348    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" event={"ID":"2a2fc86c-dde3-49c5-aa03-600f697954fd","Type":"ContainerDied","Data":"7974a1565153e052385df152e8b689569333e24d5b4d1bd8d8da1d1ffaf8a6b2"}
Nov 02 22:27:09.608494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.608417    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw" event={"ID":"2a2fc86c-dde3-49c5-aa03-600f697954fd","Type":"ContainerDied","Data":"df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7"}
Nov 02 22:27:09.608494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.608441    4929 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5b7baaa4d4b8e59ddb0a46ab419e517d858c20f0489354325d751399e563d7"
Nov 02 22:27:09.632466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.632378    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:27:09.699015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.698973    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") pod \"2a2fc86c-dde3-49c5-aa03-600f697954fd\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") "
Nov 02 22:27:09.699297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.699071    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\") pod \"2a2fc86c-dde3-49c5-aa03-600f697954fd\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") "
Nov 02 22:27:09.699297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.699160    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\" (UniqueName: \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\") pod \"2a2fc86c-dde3-49c5-aa03-600f697954fd\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") "
Nov 02 22:27:09.699297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.699190    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\") pod \"2a2fc86c-dde3-49c5-aa03-600f697954fd\" (UID: \"2a2fc86c-dde3-49c5-aa03-600f697954fd\") "
Nov 02 22:27:09.699572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.699516    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca" pod "2a2fc86c-dde3-49c5-aa03-600f697954fd" (UID: "2a2fc86c-dde3-49c5-aa03-600f697954fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:09.701462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.701436    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls" pod "2a2fc86c-dde3-49c5-aa03-600f697954fd" (UID: "2a2fc86c-dde3-49c5-aa03-600f697954fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:09.701930 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.701890    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk" pod "2a2fc86c-dde3-49c5-aa03-600f697954fd" (UID: "2a2fc86c-dde3-49c5-aa03-600f697954fd"). InnerVolumeSpecName "kube-api-access-8hvxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:09.702041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.702003    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs" pod "2a2fc86c-dde3-49c5-aa03-600f697954fd" (UID: "2a2fc86c-dde3-49c5-aa03-600f697954fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:09.800865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.800826    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2fc86c-dde3-49c5-aa03-600f697954fd-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:09.800865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.800858    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvxk\" (UniqueName: \"kubernetes.io/projected/2a2fc86c-dde3-49c5-aa03-600f697954fd-kube-api-access-8hvxk\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:09.800865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.800869    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:09.801124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:09.800877    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2fc86c-dde3-49c5-aa03-600f697954fd-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.612974 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:10.612941    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"
Nov 02 22:27:10.633914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:10.633884    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:27:10.639884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:10.639856    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-361a44-76cb85b76c-9kxmw"]
Nov 02 22:27:10.803468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:10.803388    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2fc86c-dde3-49c5-aa03-600f697954fd" path="/var/lib/kubelet/pods/2a2fc86c-dde3-49c5-aa03-600f697954fd/volumes"
Nov 02 22:27:11.769957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:11.769918    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:27:16.478565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:16.478505    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:27:16.479092 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:16.478793    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy" containerID="cri-o://2ccb8c8f7f684a4908accfee337a3e36a3cf92caeb6bbc9c14641342bcbf1c71" gracePeriod=30
Nov 02 22:27:19.773419 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:19.773380    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:27:20.632831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.632789    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:20.633404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633380    4929 cpu_manager.go:414] "containerMap: removing container" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633407    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633423    4929 cpu_manager.go:414] "containerMap: removing container" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633431    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633448    4929 cpu_manager.go:414] "containerMap: removing container" podUID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" containerName="maas-api"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633458    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" containerName="maas-api"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633472    4929 cpu_manager.go:414] "containerMap: removing container" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" containerName="main"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633480    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" containerName="main"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633492    4929 cpu_manager.go:414] "containerMap: removing container" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633500    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633513    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633521    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633547    4929 cpu_manager.go:414] "containerMap: removing container" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:20.633521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633555    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633567    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633575    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633589    4929 cpu_manager.go:414] "containerMap: removing container" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633596    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633606    4929 cpu_manager.go:414] "containerMap: removing container" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633613    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633635    4929 cpu_manager.go:414] "containerMap: removing container" podUID="f20930ed-6381-4551-8423-e9a0abfab58d" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633643    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20930ed-6381-4551-8423-e9a0abfab58d" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633658    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633665    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633675    4929 cpu_manager.go:414] "containerMap: removing container" podUID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633682    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633696    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633705    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633721    4929 cpu_manager.go:414] "containerMap: removing container" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633728    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633741    4929 cpu_manager.go:414] "containerMap: removing container" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633748    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633761    4929 cpu_manager.go:414] "containerMap: removing container" podUID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633768    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633778    4929 cpu_manager.go:414] "containerMap: removing container" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633786    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633799    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2a2fc86c-dde3-49c5-aa03-600f697954fd" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633806    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2fc86c-dde3-49c5-aa03-600f697954fd" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633819    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633826    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633837    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633845    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633858    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633865    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633876    4929 cpu_manager.go:414] "containerMap: removing container" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633883    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633895    4929 cpu_manager.go:414] "containerMap: removing container" podUID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633902    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633914    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633922    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633935    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633942    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633953    4929 cpu_manager.go:414] "containerMap: removing container" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633961    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633981    4929 cpu_manager.go:414] "containerMap: removing container" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.633988    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634003    4929 cpu_manager.go:414] "containerMap: removing container" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634010    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634021    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634029    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634039    4929 cpu_manager.go:414] "containerMap: removing container" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634047    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634060    4929 cpu_manager.go:414] "containerMap: removing container" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634069    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634080    4929 cpu_manager.go:414] "containerMap: removing container" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634087    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634105    4929 cpu_manager.go:414] "containerMap: removing container" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634113    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634124    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634131    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634144    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634151    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634163    4929 cpu_manager.go:414] "containerMap: removing container" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634171    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634184    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634192    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634203    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634211    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634222    4929 cpu_manager.go:414] "containerMap: removing container" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634229    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634240    4929 cpu_manager.go:414] "containerMap: removing container" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634248    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634260    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634267    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634280    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634288    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634297    4929 cpu_manager.go:414] "containerMap: removing container" podUID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634304    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634313    4929 cpu_manager.go:414] "containerMap: removing container" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634320    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634330    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634337    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634347    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634354    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634364    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634371    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634383    4929 cpu_manager.go:414] "containerMap: removing container" podUID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634391    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634400    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1310f3a9-262e-48fd-b6bf-703a20a036d3" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634412    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1310f3a9-262e-48fd-b6bf-703a20a036d3" containerName="maas-api"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634423    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634431    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634444    4929 cpu_manager.go:414] "containerMap: removing container" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634451    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634461    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634468    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634478    4929 cpu_manager.go:414] "containerMap: removing container" podUID="229f68f4-3a66-4375-b122-cfdf167fd1e9" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634486    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229f68f4-3a66-4375-b122-cfdf167fd1e9" containerName="payload-pre-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634497    4929 cpu_manager.go:414] "containerMap: removing container" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing"
Nov 02 22:27:20.634445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634504    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634516    4929 cpu_manager.go:414] "containerMap: removing container" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634523    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634553    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634561    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634571    4929 cpu_manager.go:414] "containerMap: removing container" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634578    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634587    4929 cpu_manager.go:414] "containerMap: removing container" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634594    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634607    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634614    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634627    4929 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634636    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634646    4929 cpu_manager.go:414] "containerMap: removing container" podUID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634652    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634760    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3637ea-9eed-432f-a7b5-0c737b3d9055" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634779    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a9d4a4-eaed-4349-8b97-1f25f09c6444" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634789    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb72af2-8e22-42c1-8633-b9ef19ca0b01" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634801    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3404cc-af7a-4aba-b97a-a62738a90624" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634812    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ff629e-baab-46c7-a7d5-d04a3f0e414d" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634824    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe82fb9-ba5d-41f2-ad57-b14d3828f3cc" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634833    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae9d89a-7ecc-4671-a704-31d4a9dc0cda" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634844    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634852    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8cb3ee-6d94-4ec9-91e8-0f8284e99c4c" containerName="maas-api"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634862    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ec59c0-fd8f-4d96-b56d-5be8c9e1633b" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634871    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2568beb3-4483-41c4-8f42-8fba1a39ea2b" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634881    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634889    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1560d752-71a3-4287-95e4-a6cf2a172c12" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634899    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634907    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3fbabf-7548-40ac-a0d1-b354f0e0e3f7" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634918    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="69074b1c-17c2-4630-9889-7fb1c7b0a8d8" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634928    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1310f3a9-262e-48fd-b6bf-703a20a036d3" containerName="maas-api"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634938    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1525ec-7195-492c-a67c-fae208e3c824" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634948    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a8b213-8821-404a-8f1b-a0cbbe32793b" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634960    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b413e10-d99a-487b-bdf5-371604909fe0" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634970    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="229f68f4-3a66-4375-b122-cfdf167fd1e9" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634980    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="616c0e90-878a-4090-8504-ecaea0c5d1d0" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634990    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="99242638-804b-46e5-ac4e-cd57dfef944f" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.634997    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635005    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="39465d60-8f30-436d-b5df-e7d25adbca88" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635014    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a017c56-f362-43c3-9efd-3461e8dcb554" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635022    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635031    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed101f72-399c-4bae-999d-68f0b1995f3a" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635042    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6f219-bc05-4252-87d1-535f3dffee4c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635049    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fe0f09-bf07-4dac-8e6d-6cfe589eb7e4" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635059    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7af1b66-40b9-4956-80f9-dc1f86cef997" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635068    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635078    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="354a47a7-8241-4d2a-a09c-dc75aaa12cd5" containerName="maas-api"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635087    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e8a816-11c1-4d18-8487-87097f71cb93" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635096    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635104    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a10639a-ee4d-4dbe-a12a-a2573ceedd24" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635113    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635120    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5776cdc7-9c34-4d5a-b6ab-6cfc19a55062" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635130    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="284396bb-975e-4e74-bb8c-e928d90b67a9" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635141    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635150    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20930ed-6381-4551-8423-e9a0abfab58d" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635158    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635168    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f36933c-5291-4a2b-9d3c-5ec3de657dd4" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635179    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8970e7-a632-48d5-9f65-d7adf860c52c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635188    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635197    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedfa64a-113a-4cdc-bcb6-5e84b818e8f9" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635207    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="99917624-eb73-4e3a-a1f3-b504df6ca6e1" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635216    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0e783b-b508-4204-bd25-c411160af26e" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635225    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2fc86c-dde3-49c5-aa03-600f697954fd" containerName="maas-api"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635234    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e46b0c-36d0-4a2e-9183-3d904bacf3bd" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635243    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635252    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d35874e-a66a-4509-8705-6f359c931155" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635261    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c177830-557e-4f11-bc68-ba22db731fda" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635270    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="43989402-9353-4abd-bf15-425b0381eaba" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635278    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e473fb-93fc-4e27-9d9e-241ea3ea9b60" containerName="payload-pre-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635288    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8f5f30-52a2-4af0-9375-128cac11fe8e" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635296    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7288e662-5c7b-4c14-8379-f3bac4ff1c2c" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635305    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="747b4185-405d-48f1-913c-fafe99cd3d82" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635315    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="772136e8-cdeb-4b26-8d15-e7cf8a1bb28e" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635324    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae75d80-8497-4240-80d7-7b55d7b97b37" containerName="main"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635333    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635341    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="123aac54-2085-49f4-8e27-483ccf274a1e" containerName="payload-processing"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635349    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635358    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb363f6-1190-4f04-8e86-1695bd76d7ee" containerName="maas-api"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635369    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="c797e29a-64d2-4f98-b179-ee4e332e00df" containerName="istio-proxy"
Nov 02 22:27:20.637592 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.635379    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e691aebd-6ff9-4836-8e7c-0f74f7bef077" containerName="main"
Nov 02 22:27:20.640288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.640267    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.658742    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.660491    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.660659    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pb6j7\""
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.661193    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.663839    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.663978    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:27:20.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.664063    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:27:20.710189 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.710145    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.710401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.710254    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.710401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.710328    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsqg\" (UniqueName: \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.811375 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.811316    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.811914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.811417    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.811914 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.811526    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsqg\" (UniqueName: \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.814668 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.814637    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.815151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.815124    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.820013 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.819982    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsqg\" (UniqueName: \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\") pod \"maas-controller-68bfb5556c-2kxts\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") " pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:20.981567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:20.981436    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:21.294611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:21.288568    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:21.314113 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:27:21.314069    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c17745_ee78_4dc8_a4a7_4c5c331e529d.slice/crio-ced4d51d553e9da5598c58bedd6dd6244d4400dd266f5de2459992069fc71849 WatchSource:0}: Error finding container ced4d51d553e9da5598c58bedd6dd6244d4400dd266f5de2459992069fc71849: Status 404 returned error can't find the container with id ced4d51d553e9da5598c58bedd6dd6244d4400dd266f5de2459992069fc71849
Nov 02 22:27:21.316580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:21.315996    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:27:21.698174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:21.697960    4929 generic.go:356] "Generic (PLEG): container finished" podID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerID="2ccb8c8f7f684a4908accfee337a3e36a3cf92caeb6bbc9c14641342bcbf1c71" exitCode=0
Nov 02 22:27:21.698174 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:21.698096    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" event={"ID":"fe0c61aa-02f5-42f0-84bd-76fb386690b9","Type":"ContainerDied","Data":"2ccb8c8f7f684a4908accfee337a3e36a3cf92caeb6bbc9c14641342bcbf1c71"}
Nov 02 22:27:21.705571 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:21.703791    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-2kxts" event={"ID":"31c17745-ee78-4dc8-a4a7-4c5c331e529d","Type":"ContainerStarted","Data":"ced4d51d553e9da5598c58bedd6dd6244d4400dd266f5de2459992069fc71849"}
Nov 02 22:27:22.025550 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.022265    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.178817    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.178892    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.178977    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179015    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179054    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179088    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179137    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179174    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179233    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\") pod \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\" (UID: \"fe0c61aa-02f5-42f0-84bd-76fb386690b9\") "
Nov 02 22:27:22.180606 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.179868    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:22.193204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.192846    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:22.205581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.200933    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:22.208557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.206692    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:22.209354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.209326    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:22.212600 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.212561    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:22.212850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.212814    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "kube-api-access-j4tpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:22.213093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.213073    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:22.215074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.215038    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo" pod "fe0c61aa-02f5-42f0-84bd-76fb386690b9" (UID: "fe0c61aa-02f5-42f0-84bd-76fb386690b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.294953    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295010    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tpf\" (UniqueName: \"kubernetes.io/projected/fe0c61aa-02f5-42f0-84bd-76fb386690b9-kube-api-access-j4tpf\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295030    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295045    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295060    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295091    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295106    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295120    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.295558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.295134    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c61aa-02f5-42f0-84bd-76fb386690b9-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.717148 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.717114    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"
Nov 02 22:27:22.717366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.717177    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9" event={"ID":"fe0c61aa-02f5-42f0-84bd-76fb386690b9","Type":"ContainerDied","Data":"3e3ca45cd4da578d0d1afea0fd04930683a0ca6e756c53c545f11d57f55bb296"}
Nov 02 22:27:22.717366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.717250    4929 scope.go:122] "RemoveContainer" containerID="2ccb8c8f7f684a4908accfee337a3e36a3cf92caeb6bbc9c14641342bcbf1c71"
Nov 02 22:27:22.760560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.741420    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:27:22.760891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.760860    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-s2qq9"]
Nov 02 22:27:22.812927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:22.812479    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" path="/var/lib/kubelet/pods/fe0c61aa-02f5-42f0-84bd-76fb386690b9/volumes"
Nov 02 22:27:25.742407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:25.742280    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-2kxts" event={"ID":"31c17745-ee78-4dc8-a4a7-4c5c331e529d","Type":"ContainerStarted","Data":"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"}
Nov 02 22:27:25.742824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:25.742510    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:25.762696 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:25.762639    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68bfb5556c-2kxts" podStartSLOduration=1.675876883 podStartE2EDuration="5.762623779s" totalImagesPullingTime="4.086746896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:25.76016591 +0000 UTC m=+1783.068256721" watchObservedRunningTime="2026-11-02 22:27:25.762623779 +0000 UTC m=+1783.070714593"
Nov 02 22:27:27.496248 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:27:27.496210    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:27:36.104214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104172    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"]
Nov 02 22:27:36.104882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104856    4929 cpu_manager.go:414] "containerMap: removing container" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:36.104882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104876    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1c6fdc-9bb4-43eb-a68b-5f1cc34ebec3" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104892    4929 cpu_manager.go:414] "containerMap: removing container" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104899    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8637bbe2-fa8d-48ef-a592-b131944fed5f" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104912    4929 cpu_manager.go:414] "containerMap: removing container" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104919    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89759db7-7ede-4cd4-9cc0-7b08206cb781" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104939    4929 cpu_manager.go:414] "containerMap: removing container" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104947    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104970    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104977    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a861bf8d-832d-4f34-8ae9-643e56c1f632" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104987    4929 cpu_manager.go:414] "containerMap: removing container" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:36.105015 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.104994    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:36.105574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.105104    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a67bb9-b21b-4a9f-ad9a-73704cae7dab" containerName="istio-proxy"
Nov 02 22:27:36.105574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.105119    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0c61aa-02f5-42f0-84bd-76fb386690b9" containerName="istio-proxy"
Nov 02 22:27:36.108744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.108717    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.124326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.124283    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"]
Nov 02 22:27:36.144392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144351    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cee094-9f9e-4b00-b658-431c6abdccc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144401    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144424    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144594    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144692    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144763    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4rm\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-kube-api-access-fb4rm\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144824    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144846    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.144964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.144945    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245721 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245678    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245739    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4rm\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-kube-api-access-fb4rm\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245766    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245783    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245827    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245849    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cee094-9f9e-4b00-b658-431c6abdccc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245877    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.245926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245908    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.245948    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.246188    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246417 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.246366    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.246414    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.246659    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.246782 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.246667    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cee094-9f9e-4b00-b658-431c6abdccc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.248223 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.248201    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.248996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.248978    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.253559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.253524    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.253681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.253665    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4rm\" (UniqueName: \"kubernetes.io/projected/e8cee094-9f9e-4b00-b658-431c6abdccc2-kube-api-access-fb4rm\") pod \"maas-default-gateway-openshift-default-696f47c6c-wjhsd\" (UID: \"e8cee094-9f9e-4b00-b658-431c6abdccc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.426333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.426244    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:36.705982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.705953    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"]
Nov 02 22:27:36.708185 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:27:36.708151    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cee094_9f9e_4b00_b658_431c6abdccc2.slice/crio-88f25b681afe18dcea34907ba6b334cab16b7c9ba218e069d0b17cd2e9a338e3 WatchSource:0}: Error finding container 88f25b681afe18dcea34907ba6b334cab16b7c9ba218e069d0b17cd2e9a338e3: Status 404 returned error can't find the container with id 88f25b681afe18dcea34907ba6b334cab16b7c9ba218e069d0b17cd2e9a338e3
Nov 02 22:27:36.710341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.710303    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:27:36.710446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.710394    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:27:36.710510 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.710442    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:27:36.750646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.750621    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:36.796400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.796345    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" event={"ID":"e8cee094-9f9e-4b00-b658-431c6abdccc2","Type":"ContainerStarted","Data":"ed32a5c7423a688dbb78be70c29cf3284bd0671baab9b2fe7556194d713e7307"}
Nov 02 22:27:36.796648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:36.796419    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" event={"ID":"e8cee094-9f9e-4b00-b658-431c6abdccc2","Type":"ContainerStarted","Data":"88f25b681afe18dcea34907ba6b334cab16b7c9ba218e069d0b17cd2e9a338e3"}
Nov 02 22:27:37.426982 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:37.426937    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:37.428781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:37.428752    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:37.428925 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:37.428803    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:38.427235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:38.427199    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:38.427672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:38.427255    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:39.427361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:39.427323    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:39.427816 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:39.427415    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:40.427653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:40.427614    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:40.428164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:40.427677    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:41.427157 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:41.427118    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:41.427361 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:41.427178    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:42.426674 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:42.426636    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:42.427083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:42.426694    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:43.427097 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:43.427052    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 22:27:43.427488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:43.427113    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 22:27:45.161738 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.161704    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:45.162210 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.162187    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:45.162973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.162950    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd"
Nov 02 22:27:45.182244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.182179    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podStartSLOduration=9.182159721 podStartE2EDuration="9.182159721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:36.815641951 +0000 UTC m=+1794.123732758" watchObservedRunningTime="2026-11-02 22:27:45.182159721 +0000 UTC m=+1802.490250527"
Nov 02 22:27:45.483041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.482950    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:27:45.575890 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.575848    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:27:45.581444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:45.581419    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:27:47.149577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:47.149529    4929 scope.go:122] "RemoveContainer" containerID="7974a1565153e052385df152e8b689569333e24d5b4d1bd8d8da1d1ffaf8a6b2"
Nov 02 22:27:55.606467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.606431    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:55.607112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.606713    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68bfb5556c-2kxts" podUID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" containerName="manager" containerID="cri-o://89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f" gracePeriod=30
Nov 02 22:27:55.856663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.856602    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:55.899730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.899690    4929 generic.go:356] "Generic (PLEG): container finished" podID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" containerID="89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f" exitCode=0
Nov 02 22:27:55.899897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.899752    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-2kxts"
Nov 02 22:27:55.899897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.899776    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-2kxts" event={"ID":"31c17745-ee78-4dc8-a4a7-4c5c331e529d","Type":"ContainerDied","Data":"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"}
Nov 02 22:27:55.899897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.899848    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-2kxts" event={"ID":"31c17745-ee78-4dc8-a4a7-4c5c331e529d","Type":"ContainerDied","Data":"ced4d51d553e9da5598c58bedd6dd6244d4400dd266f5de2459992069fc71849"}
Nov 02 22:27:55.899897 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.899883    4929 scope.go:122] "RemoveContainer" containerID="89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"
Nov 02 22:27:55.911737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.911713    4929 scope.go:122] "RemoveContainer" containerID="89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"
Nov 02 22:27:55.912095 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:27:55.912068    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f\": container with ID starting with 89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f not found: ID does not exist" containerID="89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"
Nov 02 22:27:55.912205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.912105    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f"} err="failed to get container status \"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f\": rpc error: code = NotFound desc = could not find container \"89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f\": container with ID starting with 89e249928b1eac8f2441ed9720c4eacc6dd3a317bd51e73d08d332afc959aa8f not found: ID does not exist"
Nov 02 22:27:55.954249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.954206    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\" (UniqueName: \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\") pod \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") "
Nov 02 22:27:55.954249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.954250    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\") pod \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") "
Nov 02 22:27:55.954560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.954279    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\") pod \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\" (UID: \"31c17745-ee78-4dc8-a4a7-4c5c331e529d\") "
Nov 02 22:27:55.956473 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.956439    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs" pod "31c17745-ee78-4dc8-a4a7-4c5c331e529d" (UID: "31c17745-ee78-4dc8-a4a7-4c5c331e529d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:55.956628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.956510    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert" pod "31c17745-ee78-4dc8-a4a7-4c5c331e529d" (UID: "31c17745-ee78-4dc8-a4a7-4c5c331e529d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:55.956628 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:55.956606    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg" pod "31c17745-ee78-4dc8-a4a7-4c5c331e529d" (UID: "31c17745-ee78-4dc8-a4a7-4c5c331e529d"). InnerVolumeSpecName "kube-api-access-twsqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:56.056120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.056083    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsqg\" (UniqueName: \"kubernetes.io/projected/31c17745-ee78-4dc8-a4a7-4c5c331e529d-kube-api-access-twsqg\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:56.056120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.056119    4929 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-webhook-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:56.056120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.056134    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c17745-ee78-4dc8-a4a7-4c5c331e529d-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:56.222679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.222646    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:56.226615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.226584    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-2kxts"]
Nov 02 22:27:56.802966 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:56.802932    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" path="/var/lib/kubelet/pods/31c17745-ee78-4dc8-a4a7-4c5c331e529d/volumes"
Nov 02 22:27:57.441147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.441108    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68bfb5556c-nz4xl"]
Nov 02 22:27:57.441584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.441568    4929 cpu_manager.go:414] "containerMap: removing container" podUID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" containerName="manager"
Nov 02 22:27:57.441651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.441585    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" containerName="manager"
Nov 02 22:27:57.441651 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.441648    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c17745-ee78-4dc8-a4a7-4c5c331e529d" containerName="manager"
Nov 02 22:27:57.447244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.447224    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.452195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.452168    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:27:57.452370 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.452291    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 22:27:57.452460 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.452377    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:27:57.453040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.453014    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-nz4xl"]
Nov 02 22:27:57.470435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.470399    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pb6j7\""
Nov 02 22:27:57.470638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.470447    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 22:27:57.470919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.470900    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 22:27:57.502814 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:27:57.502772    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:27:57.571602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.571516    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-metrics-certs\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.571784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.571724    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-webhook-cert\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.571784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.571747    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ph5\" (UniqueName: \"kubernetes.io/projected/7a15bfe7-9471-4d23-9fff-45014a560935-kube-api-access-b4ph5\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.672284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.672239    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-webhook-cert\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.672284 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.672290    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ph5\" (UniqueName: \"kubernetes.io/projected/7a15bfe7-9471-4d23-9fff-45014a560935-kube-api-access-b4ph5\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.672508 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.672329    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-metrics-certs\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.674952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.674921    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-webhook-cert\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.674952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.674948    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a15bfe7-9471-4d23-9fff-45014a560935-metrics-certs\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.680313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.680287    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ph5\" (UniqueName: \"kubernetes.io/projected/7a15bfe7-9471-4d23-9fff-45014a560935-kube-api-access-b4ph5\") pod \"maas-controller-68bfb5556c-nz4xl\" (UID: \"7a15bfe7-9471-4d23-9fff-45014a560935\") " pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:57.761929 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:57.761838    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:58.039120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:58.039040    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68bfb5556c-nz4xl"]
Nov 02 22:27:58.043753 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:27:58.043718    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a15bfe7_9471_4d23_9fff_45014a560935.slice/crio-de19ad8ade9403bf47cd38c0237167460a6d4d8a6b0c76e53286fca1ddca84af WatchSource:0}: Error finding container de19ad8ade9403bf47cd38c0237167460a6d4d8a6b0c76e53286fca1ddca84af: Status 404 returned error can't find the container with id de19ad8ade9403bf47cd38c0237167460a6d4d8a6b0c76e53286fca1ddca84af
Nov 02 22:27:58.919293 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:58.919256    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-nz4xl" event={"ID":"7a15bfe7-9471-4d23-9fff-45014a560935","Type":"ContainerStarted","Data":"a9d55c04584b96a2b2784fb22bc9ca8de3e49a1c9c100ca698dd4c3c6ede7199"}
Nov 02 22:27:58.919483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:58.919320    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68bfb5556c-nz4xl" event={"ID":"7a15bfe7-9471-4d23-9fff-45014a560935","Type":"ContainerStarted","Data":"de19ad8ade9403bf47cd38c0237167460a6d4d8a6b0c76e53286fca1ddca84af"}
Nov 02 22:27:58.919483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:58.919465    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:27:58.936198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:27:58.936129    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68bfb5556c-nz4xl" podStartSLOduration=1.54317137 podStartE2EDuration="1.936109614s" totalImagesPullingTime="392.938244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:58.935497454 +0000 UTC m=+1816.243588261" watchObservedRunningTime="2026-11-02 22:27:58.936109614 +0000 UTC m=+1816.244200421"
Nov 02 22:28:09.927442 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:28:09.927407    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68bfb5556c-nz4xl"
Nov 02 22:28:27.496495 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:28:27.496408    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:28:31.644030 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:28:31.643988    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:28:46.970305 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:28:46.970263    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:28:50.557567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:28:50.557460    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:28:57.495560 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:28:57.495496    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:29:05.871362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:29:05.871322    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:29:18.651688 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:29:18.651647    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:29:27.494899 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:29:27.494858    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:29:39.872970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:29:39.872928    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:29:49.979270 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:29:49.979232    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:29:53.670636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:29:53.670589    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:29:57.495642 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:29:57.495561    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:30:10.582064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:10.582025    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:14.299768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:14.299730    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:27.495451 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:30:27.495406    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:30:28.753999 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:28.753959    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:33.865559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:33.851588    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:47.451298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:47.451257    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:51.052560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:30:51.034672    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:30:57.494012 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:30:57.493974    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:31:05.151615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:05.151581    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:31:15.865676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:15.865637    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:31:16.733640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:16.733579    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:31:16.733923 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:16.733889    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery" containerID="cri-o://b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65" gracePeriod=30
Nov 02 22:31:17.066586 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.066480    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:31:17.074787 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.074752    4929 generic.go:356] "Generic (PLEG): container finished" podID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerID="b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65" exitCode=0
Nov 02 22:31:17.074956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.074799    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerDied","Data":"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"}
Nov 02 22:31:17.074956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.074845    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv" event={"ID":"0187bc2b-a254-4f14-a4cc-36b8c8be53c5","Type":"ContainerDied","Data":"d756aeda60bdf81efefd04a77ba6bc0022c1faeac242c4fee992904d62e1d454"}
Nov 02 22:31:17.074956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.074875    4929 scope.go:122] "RemoveContainer" containerID="b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"
Nov 02 22:31:17.075150 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.075039    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"
Nov 02 22:31:17.091609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.091506    4929 scope.go:122] "RemoveContainer" containerID="2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"
Nov 02 22:31:17.122770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.122415    4929 scope.go:122] "RemoveContainer" containerID="b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"
Nov 02 22:31:17.122913 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:31:17.122845    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65\": container with ID starting with b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65 not found: ID does not exist" containerID="b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"
Nov 02 22:31:17.122913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.122888    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65"} err="failed to get container status \"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65\": rpc error: code = NotFound desc = could not find container \"b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65\": container with ID starting with b2da6a7dfa2620c2a7f22d9f2567beba9a963b5ae70e2ad23935ff2ac7249b65 not found: ID does not exist"
Nov 02 22:31:17.123071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.122917    4929 scope.go:122] "RemoveContainer" containerID="2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"
Nov 02 22:31:17.123304 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:31:17.123270    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79\": container with ID starting with 2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79 not found: ID does not exist" containerID="2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"
Nov 02 22:31:17.123393 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.123314    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79"} err="failed to get container status \"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79\": rpc error: code = NotFound desc = could not find container \"2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79\": container with ID starting with 2bd93c1377e26709550e057d87c340f160337cf3569a26afab03522b14646a79 not found: ID does not exist"
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.195881    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.195951    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.196041    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.196115    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.196171    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.196206    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.196560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.196262    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\") pod \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\" (UID: \"0187bc2b-a254-4f14-a4cc-36b8c8be53c5\") "
Nov 02 22:31:17.221998 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.221786    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:17.235444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.234704    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:17.235444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.234867    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:17.235444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.234965    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:17.235444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.235089    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "kube-api-access-fsnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:17.235444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.235193    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:17.238970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.238925    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token" pod "0187bc2b-a254-4f14-a4cc-36b8c8be53c5" (UID: "0187bc2b-a254-4f14-a4cc-36b8c8be53c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297506    4929 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-kubeconfig\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297557    4929 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-cacerts\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297574    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297594    4929 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-local-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297608    4929 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-dns-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297623    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsnp5\" (UniqueName: \"kubernetes.io/projected/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-kube-api-access-fsnp5\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.297759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.297640    4929 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0187bc2b-a254-4f14-a4cc-36b8c8be53c5-istio-csr-ca-configmap\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:17.433557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.432692    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:31:17.448963 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:17.448937    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8bzv"]
Nov 02 22:31:18.821567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:18.819774    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" path="/var/lib/kubelet/pods/0187bc2b-a254-4f14-a4cc-36b8c8be53c5/volumes"
Nov 02 22:31:19.870943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:19.870902    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:31:27.495720 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:31:27.495640    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:31:40.868261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:40.868225    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:31:56.066269 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:31:56.066233    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:31:57.621054 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:31:57.621002    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:32:06.275712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:06.275677    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:32:21.681808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:21.681771    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:32:23.545558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:23.541642    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gjn6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:32:23.545558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:23.541723    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gjn6k" podUID="4a73ff87-2b7d-4c69-82e2-495f4b866345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:32:25.578956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:25.578925    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:32:27.495221 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:32:27.495185    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:32:29.492367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.492331    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:32:29.492771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.492747    4929 cpu_manager.go:414] "containerMap: removing container" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:29.492771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.492761    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:29.492857 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.492848    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:29.492892 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.492861    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:29.496004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.495986    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.504507 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.504487    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b5b20e-openshift-default-dockercfg-tb7n2\""
Nov 02 22:32:29.511338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.511313    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:32:29.581825 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.581786    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.581834    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.581906    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.581965    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kcw\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.582014    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.582044    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.582115    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.582162    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.582283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.582216    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683666    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683731    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683761    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683790    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683816    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kcw\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683856    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.683912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683890    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.684202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683926    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.684202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.683972    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.685176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.685139    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.685330 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.685290    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.685410 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.685330    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.685480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.685400    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.685480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.685431    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.686815 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.686793    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.686944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.686928    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.692643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.692615    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.692773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.692756    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kcw\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\") pod \"e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") " pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:29.809362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:29.809281    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:30.091409 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.091379    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:32:30.094234 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:32:30.094205    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a77f90a_bc67_4bb3_bfad_64aa7a94c4b1.slice/crio-f76d9eacce72f968ab59f1e8cddd17e8bddd20754cddac33886285a831272bcd WatchSource:0}: Error finding container f76d9eacce72f968ab59f1e8cddd17e8bddd20754cddac33886285a831272bcd: Status 404 returned error can't find the container with id f76d9eacce72f968ab59f1e8cddd17e8bddd20754cddac33886285a831272bcd
Nov 02 22:32:30.096424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.096388    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:30.096569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.096466    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:30.096569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.096509    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:30.638111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.638064    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerStarted","Data":"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"}
Nov 02 22:32:30.638518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.638130    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerStarted","Data":"f76d9eacce72f968ab59f1e8cddd17e8bddd20754cddac33886285a831272bcd"}
Nov 02 22:32:30.661180 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.661122    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podStartSLOduration=1.661106247 podStartE2EDuration="1.661106247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:30.658315207 +0000 UTC m=+2087.966406011" watchObservedRunningTime="2026-11-02 22:32:30.661106247 +0000 UTC m=+2087.969197044"
Nov 02 22:32:30.809990 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:30.809955    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:31.814554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:31.814508    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:32.647250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:32.647216    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:32.648308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:32.648284    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:34.031318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.031280    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:32:34.031870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.031848    4929 cpu_manager.go:414] "containerMap: removing container" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:34.031870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.031870    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187bc2b-a254-4f14-a4cc-36b8c8be53c5" containerName="discovery"
Nov 02 22:32:34.035805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.035756    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.047602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.047566    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b5b20e-serving-cert\""
Nov 02 22:32:34.054021 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.053996    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:32:34.057707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.057683    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:32:34.061569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.061517    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.067898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.067871    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:32:34.068203 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.068174    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b5b20e\""
Nov 02 22:32:34.090463 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.090433    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:32:34.093960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.093937    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.107125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.107094    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:32:34.130589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130551    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdh2k\" (UniqueName: \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.130770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130604    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.130770 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130635    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.130889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130777    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknnt\" (UniqueName: \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.130889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130833    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.130889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130870    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.131001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130927    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.131001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.130948    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tzh\" (UniqueName: \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.146529 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.146505    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-febfe0a3-dockercfg-z7vtp\""
Nov 02 22:32:34.231913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.231852    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.231913 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.231910    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tzh\" (UniqueName: \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.232297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.231967    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdh2k\" (UniqueName: \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.232297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.231993    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.232297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.232030    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.232887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.232858    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.239383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.233069    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknnt\" (UniqueName: \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.239383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.233123    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.239383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.233158    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.239383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.236229    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.239383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.236672    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.241887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.241841    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.242286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.242260    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.244922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.244881    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknnt\" (UniqueName: \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\") pod \"payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.248237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.248197    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdh2k\" (UniqueName: \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\") pod \"payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.254163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.254128    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tzh\" (UniqueName: \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\") pod \"maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.362025 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.361995    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:34.378969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.378935    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:34.409233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.409197    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:34.644981 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.644891    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:32:34.648156 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:32:34.648120    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfd8309_ad20_4346_b8d5_4ea16f9a4898.slice/crio-7dcc62139647968632fbcd32574c413b9d1f242896c1e4e35d76370ec8493a0b WatchSource:0}: Error finding container 7dcc62139647968632fbcd32574c413b9d1f242896c1e4e35d76370ec8493a0b: Status 404 returned error can't find the container with id 7dcc62139647968632fbcd32574c413b9d1f242896c1e4e35d76370ec8493a0b
Nov 02 22:32:34.649679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.649657    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:32:34.657774 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.657733    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" event={"ID":"9bfd8309-ad20-4346-b8d5-4ea16f9a4898","Type":"ContainerStarted","Data":"7dcc62139647968632fbcd32574c413b9d1f242896c1e4e35d76370ec8493a0b"}
Nov 02 22:32:34.671863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.671831    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:32:34.673575 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:32:34.673550    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee17d02_74d5_4ab1_b942_f177004e4f4f.slice/crio-f5a6de5635b44f4035418513e305cbade2fec0a8995834ec98a20f8bc6791b04 WatchSource:0}: Error finding container f5a6de5635b44f4035418513e305cbade2fec0a8995834ec98a20f8bc6791b04: Status 404 returned error can't find the container with id f5a6de5635b44f4035418513e305cbade2fec0a8995834ec98a20f8bc6791b04
Nov 02 22:32:34.713755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:34.713719    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:32:34.729799 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:32:34.729769    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48fff75_83bd_4955_8037_72a437e84779.slice/crio-11d4bfcbcdaa323a090b94b3eca30245a64125d5d20c6918c4064a4151869e5d WatchSource:0}: Error finding container 11d4bfcbcdaa323a090b94b3eca30245a64125d5d20c6918c4064a4151869e5d: Status 404 returned error can't find the container with id 11d4bfcbcdaa323a090b94b3eca30245a64125d5d20c6918c4064a4151869e5d
Nov 02 22:32:35.664605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.664559    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" event={"ID":"9bfd8309-ad20-4346-b8d5-4ea16f9a4898","Type":"ContainerStarted","Data":"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"}
Nov 02 22:32:35.665126 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.664773    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:35.666209 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.666183    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" event={"ID":"c48fff75-83bd-4955-8037-72a437e84779","Type":"ContainerStarted","Data":"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"}
Nov 02 22:32:35.666307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.666230    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" event={"ID":"c48fff75-83bd-4955-8037-72a437e84779","Type":"ContainerStarted","Data":"11d4bfcbcdaa323a090b94b3eca30245a64125d5d20c6918c4064a4151869e5d"}
Nov 02 22:32:35.666366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.666316    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:35.667808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.667782    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" event={"ID":"7ee17d02-74d5-4ab1-b942-f177004e4f4f","Type":"ContainerStarted","Data":"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"}
Nov 02 22:32:35.667934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.667827    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" event={"ID":"7ee17d02-74d5-4ab1-b942-f177004e4f4f","Type":"ContainerStarted","Data":"f5a6de5635b44f4035418513e305cbade2fec0a8995834ec98a20f8bc6791b04"}
Nov 02 22:32:35.667934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.667852    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:35.683125 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.683070    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" podStartSLOduration=1.233321618 podStartE2EDuration="1.683032385s" totalImagesPullingTime="449.710767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:35.681972217 +0000 UTC m=+2092.990063022" watchObservedRunningTime="2026-11-02 22:32:35.683032385 +0000 UTC m=+2092.991123190"
Nov 02 22:32:35.701516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.701459    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" podStartSLOduration=1.701443125 podStartE2EDuration="1.701443125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:35.699666948 +0000 UTC m=+2093.007757752" watchObservedRunningTime="2026-11-02 22:32:35.701443125 +0000 UTC m=+2093.009533930"
Nov 02 22:32:35.714953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:35.714893    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" podStartSLOduration=1.714878134 podStartE2EDuration="1.714878134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:35.714564048 +0000 UTC m=+2093.022654855" watchObservedRunningTime="2026-11-02 22:32:35.714878134 +0000 UTC m=+2093.022968939"
Nov 02 22:32:36.285594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:36.285553    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:32:41.686585 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:41.686553    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:32:45.609870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:45.609835    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:32:45.615198 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:45.615170    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:32:46.684008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.683979    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:32:46.684476 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.684045    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:32:46.740669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.740627    4929 generic.go:356] "Generic (PLEG): container finished" podID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerID="dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786" exitCode=0
Nov 02 22:32:46.740839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.740693    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerDied","Data":"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"}
Nov 02 22:32:46.741215 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.741190    4929 scope.go:122] "RemoveContainer" containerID="dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"
Nov 02 22:32:46.742605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.742574    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:46.742713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.742661    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:46.742776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:46.742712    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:32:47.652701 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:47.652652    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:47.766485 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:47.766439    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerStarted","Data":"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"}
Nov 02 22:32:47.810411 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:47.810380    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:47.811437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:47.811407    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 22:32:47.811510 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:47.811455    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 22:32:48.810081 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:48.810047    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 22:32:48.810474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:48.810127    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 22:32:49.722703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.722670    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:32:49.726590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.726567    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.734332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.733814    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b5b20e\"/\"default-dockercfg-6rlr9\""
Nov 02 22:32:49.764205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.764170    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:32:49.765922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.765893    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b5b20e\"/\"simulator-single-main-b5b20e-kserve-self-signed-certs\""
Nov 02 22:32:49.766192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.766172    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b5b20e\"/\"kube-root-ca.crt\""
Nov 02 22:32:49.766481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.766434    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b5b20e\"/\"openshift-service-ca.crt\""
Nov 02 22:32:49.795461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795430    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.795647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795475    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.795647 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795618    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.795777 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795687    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.795840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795801    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr678\" (UniqueName: \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.795902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.795852    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.810423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.810398    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 22:32:49.810827 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.810439    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 22:32:49.896686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.896643    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.896909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.896722    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr678\" (UniqueName: \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.896909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.896765    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.896909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.896800    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.896909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.896874    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.897250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.897002    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.897317 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.897255    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.899357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.898747    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.900049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.900028    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.901121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.901092    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.901861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.901839    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:49.906008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:49.905959    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr678\" (UniqueName: \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\") pod \"simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") " pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:50.072504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.072408    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:50.361120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.361087    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:32:50.364403 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:32:50.364371    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode05fcba6_f5d7_4b53_898b_9fb3aab6abe5.slice/crio-c7543bf409122bd67b903274db227039c31e5fde69f4f94c285773e2618a528b WatchSource:0}: Error finding container c7543bf409122bd67b903274db227039c31e5fde69f4f94c285773e2618a528b: Status 404 returned error can't find the container with id c7543bf409122bd67b903274db227039c31e5fde69f4f94c285773e2618a528b
Nov 02 22:32:50.783791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.783699    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" event={"ID":"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5","Type":"ContainerStarted","Data":"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"}
Nov 02 22:32:50.783791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.783747    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" event={"ID":"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5","Type":"ContainerStarted","Data":"c7543bf409122bd67b903274db227039c31e5fde69f4f94c285773e2618a528b"}
Nov 02 22:32:50.800390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.800337    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" podStartSLOduration=1.800322567 podStartE2EDuration="1.800322567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:50.798806647 +0000 UTC m=+2108.106897456" watchObservedRunningTime="2026-11-02 22:32:50.800322567 +0000 UTC m=+2108.108413372"
Nov 02 22:32:50.810122 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.810089    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 22:32:50.810294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:50.810136    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 22:32:51.073622 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:51.073514    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:32:51.362869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:51.362823    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:32:51.809962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:51.809873    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 22:32:51.809962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:51.809932    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 22:32:53.179685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:53.179651    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:53.180219 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:53.180039    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:53.181137 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:32:53.181112    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:32:57.496024 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:32:57.495938    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:33:00.072988 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:00.072952    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:33:00.085755 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:00.085728    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:33:00.843246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:00.843214    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:33:07.178952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:07.178916    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:33:27.496115 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:27.496071    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:33:31.257217 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:31.257179    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:33:35.619271 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:35.619215    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:33:40.894700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:40.894662    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:33:40.908714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:40.908683    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:33:41.702469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:41.702428    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:33:41.702713 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:41.702674    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" podUID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" containerName="maas-api" containerID="cri-o://d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae" gracePeriod=30
Nov 02 22:33:41.953767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:41.953703    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:33:42.040865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.040758    4929 generic.go:356] "Generic (PLEG): container finished" podID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" containerID="d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae" exitCode=0
Nov 02 22:33:42.041762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.041134    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" event={"ID":"9bfd8309-ad20-4346-b8d5-4ea16f9a4898","Type":"ContainerDied","Data":"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"}
Nov 02 22:33:42.041762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.041197    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk" event={"ID":"9bfd8309-ad20-4346-b8d5-4ea16f9a4898","Type":"ContainerDied","Data":"7dcc62139647968632fbcd32574c413b9d1f242896c1e4e35d76370ec8493a0b"}
Nov 02 22:33:42.041762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.041227    4929 scope.go:122] "RemoveContainer" containerID="d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"
Nov 02 22:33:42.041762 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.041392    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"
Nov 02 22:33:42.053001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.052971    4929 scope.go:122] "RemoveContainer" containerID="d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"
Nov 02 22:33:42.053354 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:42.053331    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae\": container with ID starting with d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae not found: ID does not exist" containerID="d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"
Nov 02 22:33:42.053448 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.053364    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae"} err="failed to get container status \"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae\": rpc error: code = NotFound desc = could not find container \"d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae\": container with ID starting with d64bfdd634e1961fed49aa7cbdc80dd3b9a78de3707324c9832c5b0fc1a1bcae not found: ID does not exist"
Nov 02 22:33:42.137776 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.137739    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\") pod \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") "
Nov 02 22:33:42.137973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.137868    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\") pod \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") "
Nov 02 22:33:42.137973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.137919    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\") pod \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") "
Nov 02 22:33:42.137973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.137948    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\" (UniqueName: \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\") pod \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\" (UID: \"9bfd8309-ad20-4346-b8d5-4ea16f9a4898\") "
Nov 02 22:33:42.138167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.138139    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca" pod "9bfd8309-ad20-4346-b8d5-4ea16f9a4898" (UID: "9bfd8309-ad20-4346-b8d5-4ea16f9a4898"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:33:42.138336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.138311    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:42.140108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.140070    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls" pod "9bfd8309-ad20-4346-b8d5-4ea16f9a4898" (UID: "9bfd8309-ad20-4346-b8d5-4ea16f9a4898"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:33:42.140201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.140102    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs" pod "9bfd8309-ad20-4346-b8d5-4ea16f9a4898" (UID: "9bfd8309-ad20-4346-b8d5-4ea16f9a4898"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:33:42.140201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.140131    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh" pod "9bfd8309-ad20-4346-b8d5-4ea16f9a4898" (UID: "9bfd8309-ad20-4346-b8d5-4ea16f9a4898"). InnerVolumeSpecName "kube-api-access-l2tzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:33:42.240001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.239912    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:42.240001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.239950    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:42.240001 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.239966    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tzh\" (UniqueName: \"kubernetes.io/projected/9bfd8309-ad20-4346-b8d5-4ea16f9a4898-kube-api-access-l2tzh\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:42.363095 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.363052    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:33:42.368612 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.368586    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b5b20e-5d7c5bb64-4d7vk"]
Nov 02 22:33:42.802818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:42.802785    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" path="/var/lib/kubelet/pods/9bfd8309-ad20-4346-b8d5-4ea16f9a4898/volumes"
Nov 02 22:33:43.546052 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:43.546004    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:33:45.895360 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:45.895319    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing" containerID="cri-o://6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a" gracePeriod=25
Nov 02 22:33:45.909468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:45.909425    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing" containerID="cri-o://2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d" gracePeriod=25
Nov 02 22:33:46.681781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:46.681739    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:46.681781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:46.681760    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:48.343645 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:48.343606    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:33:48.344051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:48.343915    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" containerID="cri-o://bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3" gracePeriod=30
Nov 02 22:33:50.639908 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.639835    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerName="main" containerID="cri-o://4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd" gracePeriod=45
Nov 02 22:33:50.893042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.892975    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:33:50.918006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.917970    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918095    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918163    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918210    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918238    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918247    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:50.918495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918276    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\" (UniqueName: \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\") pod \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\" (UID: \"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5\") "
Nov 02 22:33:50.918495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918448    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:50.918683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918628    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:50.918683 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918649    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:50.918870 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.918846    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:50.921119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.921078    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:50.921235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.921128    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:33:50.921235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:50.921166    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678" pod "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" (UID: "e05fcba6-f5d7-4b53-898b-9fb3aab6abe5"). InnerVolumeSpecName "kube-api-access-xr678". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:33:51.019970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.019936    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:51.019970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.019965    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:51.019970 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.019976    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr678\" (UniqueName: \"kubernetes.io/projected/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-kube-api-access-xr678\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:51.020201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.019986    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:51.084116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.084079    4929 generic.go:356] "Generic (PLEG): container finished" podID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerID="4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd" exitCode=0
Nov 02 22:33:51.084300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.084124    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"
Nov 02 22:33:51.084300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.084166    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" event={"ID":"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5","Type":"ContainerDied","Data":"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"}
Nov 02 22:33:51.084300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.084209    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" event={"ID":"e05fcba6-f5d7-4b53-898b-9fb3aab6abe5","Type":"ContainerDied","Data":"c7543bf409122bd67b903274db227039c31e5fde69f4f94c285773e2618a528b"}
Nov 02 22:33:51.084300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.084227    4929 scope.go:122] "RemoveContainer" containerID="4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"
Nov 02 22:33:51.093424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.093400    4929 scope.go:122] "RemoveContainer" containerID="4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"
Nov 02 22:33:51.093753 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:51.093730    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd\": container with ID starting with 4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd not found: ID does not exist" containerID="4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"
Nov 02 22:33:51.093823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.093761    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd"} err="failed to get container status \"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd\": rpc error: code = NotFound desc = could not find container \"4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd\": container with ID starting with 4e6d050712bef0ac7eec79f61e76ea3582c69d4cbd96d9f80cd2494b98e508fd not found: ID does not exist"
Nov 02 22:33:51.105014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.104993    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:33:51.110433 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.110411    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z"]
Nov 02 22:33:51.832944 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:51.832899    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-b5b20e/simulator-single-main-b5b20e-kserve-55c98c6d8d-2zp6z" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:33:52.803115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:52.803085    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" path="/var/lib/kubelet/pods/e05fcba6-f5d7-4b53-898b-9fb3aab6abe5/volumes"
Nov 02 22:33:53.183288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:53.183252    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:33:53.183675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:53.183309    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:33:54.003195 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.003170    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:33:54.046860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046831    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.046860 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046865    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046894    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046929    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046948    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.046995    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047045    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047104 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047069    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047152    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\") pod \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\" (UID: \"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1\") "
Nov 02 22:33:54.047424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047189    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:54.047424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047207    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:54.047424 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047389    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:54.047677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047401    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:54.047677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047451    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.047677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047471    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.047677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.047590    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:33:54.049141 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.049118    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:33:54.049372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.049356    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:33:54.049676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.049648    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:33:54.049805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.049786    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw" pod "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" (UID: "5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1"). InnerVolumeSpecName "kube-api-access-x4kcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:33:54.099855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.099821    4929 generic.go:356] "Generic (PLEG): container finished" podID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerID="bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3" exitCode=0
Nov 02 22:33:54.100023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.099859    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerDied","Data":"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"}
Nov 02 22:33:54.100023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.099874    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"
Nov 02 22:33:54.100023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.099901    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw" event={"ID":"5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1","Type":"ContainerDied","Data":"f76d9eacce72f968ab59f1e8cddd17e8bddd20754cddac33886285a831272bcd"}
Nov 02 22:33:54.100023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.099923    4929 scope.go:122] "RemoveContainer" containerID="bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"
Nov 02 22:33:54.109120 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.109100    4929 scope.go:122] "RemoveContainer" containerID="dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"
Nov 02 22:33:54.117136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.117115    4929 scope.go:122] "RemoveContainer" containerID="bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"
Nov 02 22:33:54.117390 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:54.117370    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3\": container with ID starting with bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3 not found: ID does not exist" containerID="bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"
Nov 02 22:33:54.117466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.117399    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3"} err="failed to get container status \"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3\": rpc error: code = NotFound desc = could not find container \"bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3\": container with ID starting with bcbcab580216aae1bb475d4f7a0d21f86cf4dead0ddf1cda7cd4e062fbade3c3 not found: ID does not exist"
Nov 02 22:33:54.117466 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.117416    4929 scope.go:122] "RemoveContainer" containerID="dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"
Nov 02 22:33:54.118277 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:54.118258    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786\": container with ID starting with dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786 not found: ID does not exist" containerID="dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"
Nov 02 22:33:54.118328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.118283    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786"} err="failed to get container status \"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786\": rpc error: code = NotFound desc = could not find container \"dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786\": container with ID starting with dabff520eb79b6d1d4519e9412be67d6ac91ca4a7745e1a85c6e8eecc1912786 not found: ID does not exist"
Nov 02 22:33:54.122151 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.122128    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:33:54.125783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.125761    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b5b20e-openshift-default-759996dbb5z25dw"]
Nov 02 22:33:54.147956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147928    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.147956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147954    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.148074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147964    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.148074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147972    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.148074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147980    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.148074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147989    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kcw\" (UniqueName: \"kubernetes.io/projected/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-kube-api-access-x4kcw\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.148074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.147998    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:54.803162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:54.803128    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" path="/var/lib/kubelet/pods/5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1/volumes"
Nov 02 22:33:56.682236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:56.682186    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:56.682698 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:33:56.682199    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:57.494839 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:33:57.494797    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:34:00.559742 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:00.559700    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:34:04.479164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:04.479126    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:34:06.168736 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.168666    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:34:06.198748 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.198719    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:34:06.198921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.198797    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\") pod \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") "
Nov 02 22:34:06.198921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.198827    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\" (UniqueName: \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\") pod \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\" (UID: \"7ee17d02-74d5-4ab1-b942-f177004e4f4f\") "
Nov 02 22:34:06.199253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.199197    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume" pod "7ee17d02-74d5-4ab1-b942-f177004e4f4f" (UID: "7ee17d02-74d5-4ab1-b942-f177004e4f4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:06.200863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.200838    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt" pod "7ee17d02-74d5-4ab1-b942-f177004e4f4f" (UID: "7ee17d02-74d5-4ab1-b942-f177004e4f4f"). InnerVolumeSpecName "kube-api-access-qknnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:06.210112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.210080    4929 generic.go:356] "Generic (PLEG): container finished" podID="c48fff75-83bd-4955-8037-72a437e84779" containerID="2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d" exitCode=0
Nov 02 22:34:06.210249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.210134    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"
Nov 02 22:34:06.210249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.210161    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" event={"ID":"c48fff75-83bd-4955-8037-72a437e84779","Type":"ContainerDied","Data":"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"}
Nov 02 22:34:06.210249 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.210220    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6" event={"ID":"c48fff75-83bd-4955-8037-72a437e84779","Type":"ContainerDied","Data":"11d4bfcbcdaa323a090b94b3eca30245a64125d5d20c6918c4064a4151869e5d"}
Nov 02 22:34:06.210431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.210254    4929 scope.go:122] "RemoveContainer" containerID="2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"
Nov 02 22:34:06.211610 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.211589    4929 generic.go:356] "Generic (PLEG): container finished" podID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerID="6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a" exitCode=0
Nov 02 22:34:06.211724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.211638    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"
Nov 02 22:34:06.211724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.211672    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" event={"ID":"7ee17d02-74d5-4ab1-b942-f177004e4f4f","Type":"ContainerDied","Data":"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"}
Nov 02 22:34:06.211849 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.211736    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf" event={"ID":"7ee17d02-74d5-4ab1-b942-f177004e4f4f","Type":"ContainerDied","Data":"f5a6de5635b44f4035418513e305cbade2fec0a8995834ec98a20f8bc6791b04"}
Nov 02 22:34:06.221432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.221294    4929 scope.go:122] "RemoveContainer" containerID="2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"
Nov 02 22:34:06.221905 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:34:06.221872    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d\": container with ID starting with 2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d not found: ID does not exist" containerID="2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"
Nov 02 22:34:06.222007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.221910    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d"} err="failed to get container status \"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d\": rpc error: code = NotFound desc = could not find container \"2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d\": container with ID starting with 2beeefe9d4c1cc2a839551d2dbb2c814a59759b1b767ef136d8836ade3fd1c7d not found: ID does not exist"
Nov 02 22:34:06.222007 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.221932    4929 scope.go:122] "RemoveContainer" containerID="6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"
Nov 02 22:34:06.232297 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.232277    4929 scope.go:122] "RemoveContainer" containerID="6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"
Nov 02 22:34:06.232630 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:34:06.232609    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a\": container with ID starting with 6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a not found: ID does not exist" containerID="6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"
Nov 02 22:34:06.232730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.232635    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a"} err="failed to get container status \"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a\": rpc error: code = NotFound desc = could not find container \"6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a\": container with ID starting with 6301d2addf301608011256798109472cf6727801d0ff45d8dc325d0ec7a3b72a not found: ID does not exist"
Nov 02 22:34:06.236607 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.236584    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:34:06.241658 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.241620    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b5b20e-6769766b7q55lf"]
Nov 02 22:34:06.299627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.299595    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\") pod \"c48fff75-83bd-4955-8037-72a437e84779\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") "
Nov 02 22:34:06.299792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.299756    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\" (UniqueName: \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\") pod \"c48fff75-83bd-4955-8037-72a437e84779\" (UID: \"c48fff75-83bd-4955-8037-72a437e84779\") "
Nov 02 22:34:06.300056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.300036    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume" pod "c48fff75-83bd-4955-8037-72a437e84779" (UID: "c48fff75-83bd-4955-8037-72a437e84779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:06.300094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.300044    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee17d02-74d5-4ab1-b942-f177004e4f4f-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:06.300094 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.300083    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknnt\" (UniqueName: \"kubernetes.io/projected/7ee17d02-74d5-4ab1-b942-f177004e4f4f-kube-api-access-qknnt\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:06.301793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.301772    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k" pod "c48fff75-83bd-4955-8037-72a437e84779" (UID: "c48fff75-83bd-4955-8037-72a437e84779"). InnerVolumeSpecName "kube-api-access-gdh2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:06.400910 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.400874    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48fff75-83bd-4955-8037-72a437e84779-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:06.400910 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.400905    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdh2k\" (UniqueName: \"kubernetes.io/projected/c48fff75-83bd-4955-8037-72a437e84779-kube-api-access-gdh2k\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:06.530155 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.530126    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:34:06.536049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.536025    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b5b20e-566d5wxcd6"]
Nov 02 22:34:06.803504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.803425    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" path="/var/lib/kubelet/pods/7ee17d02-74d5-4ab1-b942-f177004e4f4f/volumes"
Nov 02 22:34:06.803823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:34:06.803809    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48fff75-83bd-4955-8037-72a437e84779" path="/var/lib/kubelet/pods/c48fff75-83bd-4955-8037-72a437e84779/volumes"
Nov 02 22:34:27.495319 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:34:27.495234    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:34:57.496112 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:34:57.496068    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:35:27.442054 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:27.442015    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:35:27.495750 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:35:27.495713    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:35:42.079646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:42.079608    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:35:45.987590 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:45.987551    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:35:46.233557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:46.232672    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wjhsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:35:46.233557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:46.232850    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wjhsd" podUID="e8cee094-9f9e-4b00-b658-431c6abdccc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:35:46.316751 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:46.316645    4929 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-2zhcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:35:46.317034 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:35:46.317001    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-2zhcx" podUID="5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:35:57.495226 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:35:57.495132    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:36:01.093563 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:01.093514    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:36:02.301987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.301947    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:36:02.302614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302592    4929 cpu_manager.go:414] "containerMap: removing container" podUID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" containerName="maas-api"
Nov 02 22:36:02.302686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302618    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" containerName="maas-api"
Nov 02 22:36:02.302686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302663    4929 cpu_manager.go:414] "containerMap: removing container" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing"
Nov 02 22:36:02.302686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302672    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing"
Nov 02 22:36:02.302686 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302682    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302691    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302719    4929 cpu_manager.go:414] "containerMap: removing container" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302727    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302750    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerName="main"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302757    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerName="main"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302864    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee17d02-74d5-4ab1-b942-f177004e4f4f" containerName="payload-processing"
Nov 02 22:36:02.302885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302878    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:02.303204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302890    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48fff75-83bd-4955-8037-72a437e84779" containerName="payload-pre-processing"
Nov 02 22:36:02.303204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302905    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05fcba6-f5d7-4b53-898b-9fb3aab6abe5" containerName="main"
Nov 02 22:36:02.303204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302914    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:02.303204 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.302924    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfd8309-ad20-4346-b8d5-4ea16f9a4898" containerName="maas-api"
Nov 02 22:36:02.306692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.306663    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.314953 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.314926    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-93446a-openshift-default-dockercfg-62j8d\""
Nov 02 22:36:02.318832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.318804    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:36:02.398614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398573    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398627    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398725    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398804 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398743    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398827    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398865    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.398917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398908    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdfc\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.399014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398952    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.399014 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.398990    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500057 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500022    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500066    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500088    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500166    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500221    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500306    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500376    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500401    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500479    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdfc\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500504    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500590    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.500932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.500877    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.501119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.501097    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.501202 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.501138    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.504159 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.504131    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.504328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.504304    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.507746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.507722    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.508197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.508175    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdfc\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\") pod \"e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") " pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.626404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.626366    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:02.907526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.907502    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:36:02.909725 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:36:02.909701    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d480a9_7703_447f_9863_6e877ba1c7f4.slice/crio-515fcf23d27fb84ab11fc26be94a2d185c6812f761d67d0abdab22a28455120a WatchSource:0}: Error finding container 515fcf23d27fb84ab11fc26be94a2d185c6812f761d67d0abdab22a28455120a: Status 404 returned error can't find the container with id 515fcf23d27fb84ab11fc26be94a2d185c6812f761d67d0abdab22a28455120a
Nov 02 22:36:02.912225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.912189    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:02.912337 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.912268    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:02.912337 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:02.912313    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:03.781115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:03.781077    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerStarted","Data":"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"}
Nov 02 22:36:03.781523 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:03.781155    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerStarted","Data":"515fcf23d27fb84ab11fc26be94a2d185c6812f761d67d0abdab22a28455120a"}
Nov 02 22:36:03.801153 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:03.801098    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podStartSLOduration=1.8010819310000001 podStartE2EDuration="1.801081931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:03.799179745 +0000 UTC m=+2301.107270542" watchObservedRunningTime="2026-11-02 22:36:03.801081931 +0000 UTC m=+2301.109172738"
Nov 02 22:36:04.627234 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:04.627197    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:04.632283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:04.632257    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:04.785591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:04.785558    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:04.786812 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:04.786792    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:07.132333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.132292    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:36:07.132931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.132783    4929 cpu_manager.go:414] "containerMap: removing container" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:07.132931 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.132799    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a77f90a-bc67-4bb3-bfad-64aa7a94c4b1" containerName="istio-proxy"
Nov 02 22:36:07.136358 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.136327    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.145276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.145243    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-93446a-serving-cert\""
Nov 02 22:36:07.149840 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.149693    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:36:07.207225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.207189    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:36:07.217670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.217393    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.227414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.227384    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:36:07.234694 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.234651    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-93446a\""
Nov 02 22:36:07.245808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.245772    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.245985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.245833    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9r7g\" (UniqueName: \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.245985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.245917    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.246100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.245990    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2ht\" (UniqueName: \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.246136 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.246088    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.246169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.246157    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.303841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.303805    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:36:07.307350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.307330    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.320245 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.320218    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-31b46b88-dockercfg-ft8q8\""
Nov 02 22:36:07.347006 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.346968    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.347176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347048    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47kjg\" (UniqueName: \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.347176 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347082    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.347280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347165    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9r7g\" (UniqueName: \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.347280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347199    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.347280 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347263    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2ht\" (UniqueName: \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.347494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347467    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.347616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.347567    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.348308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.348271    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.348481 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.348461    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.350577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.350551    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.350841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.350818    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:36:07.350920 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.350891    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.368426 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.368394    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9r7g\" (UniqueName: \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\") pod \"payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.368603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.368512    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2ht\" (UniqueName: \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\") pod \"maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.449602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.449451    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47kjg\" (UniqueName: \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.449799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.449615    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.450349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.450326    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.454382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.454349    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:07.457975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.457950    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47kjg\" (UniqueName: \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\") pod \"payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.533294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.533255    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:07.620919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.620882    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:07.742467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.741434    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:36:07.800082 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.800048    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" event={"ID":"59933d41-ed50-4532-8f74-ca6ec99d4e0b","Type":"ContainerStarted","Data":"b25f9e005279fb9ab47a9482e9ee65004d86a3574eed68e7b00cad9833184efc"}
Nov 02 22:36:07.811448 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:36:07.811422    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e15a015_048e_4711_b7bc_a955814f2662.slice/crio-74c5f2f2f176cf79c7eaa578ad141a5dcd4122c51b97e05347e18e2f0bb51933 WatchSource:0}: Error finding container 74c5f2f2f176cf79c7eaa578ad141a5dcd4122c51b97e05347e18e2f0bb51933: Status 404 returned error can't find the container with id 74c5f2f2f176cf79c7eaa578ad141a5dcd4122c51b97e05347e18e2f0bb51933
Nov 02 22:36:07.812170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.812144    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:36:07.900422 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:07.900396    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:36:07.902583 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:36:07.902554    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592e6358_00fc_4e55_88b5_746a918e54d2.slice/crio-d14853f251344a50ca3fc72fbf9358e1f4fcfc8df83d2965af74da3f679ecd8f WatchSource:0}: Error finding container d14853f251344a50ca3fc72fbf9358e1f4fcfc8df83d2965af74da3f679ecd8f: Status 404 returned error can't find the container with id d14853f251344a50ca3fc72fbf9358e1f4fcfc8df83d2965af74da3f679ecd8f
Nov 02 22:36:08.806020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.805978    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" event={"ID":"59933d41-ed50-4532-8f74-ca6ec99d4e0b","Type":"ContainerStarted","Data":"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"}
Nov 02 22:36:08.806499 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.806056    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:08.807567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.807506    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" event={"ID":"592e6358-00fc-4e55-88b5-746a918e54d2","Type":"ContainerStarted","Data":"d3fc2a69ece4d866b565f7ed19f349af2fae5961f8f1b1df4ac0333b22009d88"}
Nov 02 22:36:08.807711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.807566    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" event={"ID":"592e6358-00fc-4e55-88b5-746a918e54d2","Type":"ContainerStarted","Data":"d14853f251344a50ca3fc72fbf9358e1f4fcfc8df83d2965af74da3f679ecd8f"}
Nov 02 22:36:08.807711 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.807597    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:08.808917 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.808891    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" event={"ID":"8e15a015-048e-4711-b7bc-a955814f2662","Type":"ContainerStarted","Data":"21ad4047c0166012443007dc4cc3fbcf26446597f56009fde5a042ef90d71aea"}
Nov 02 22:36:08.809020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.808931    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" event={"ID":"8e15a015-048e-4711-b7bc-a955814f2662","Type":"ContainerStarted","Data":"74c5f2f2f176cf79c7eaa578ad141a5dcd4122c51b97e05347e18e2f0bb51933"}
Nov 02 22:36:08.809020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.808995    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:08.827160 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.827107    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" podStartSLOduration=1.396813327 podStartE2EDuration="1.827091448s" totalImagesPullingTime="430.278121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:08.825716018 +0000 UTC m=+2306.133806823" watchObservedRunningTime="2026-11-02 22:36:08.827091448 +0000 UTC m=+2306.135182252"
Nov 02 22:36:08.840437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.840386    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" podStartSLOduration=1.840372322 podStartE2EDuration="1.840372322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:08.839670888 +0000 UTC m=+2306.147761693" watchObservedRunningTime="2026-11-02 22:36:08.840372322 +0000 UTC m=+2306.148463126"
Nov 02 22:36:08.854794 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.854740    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" podStartSLOduration=1.854727155 podStartE2EDuration="1.854727155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:08.85405411 +0000 UTC m=+2306.162144913" watchObservedRunningTime="2026-11-02 22:36:08.854727155 +0000 UTC m=+2306.162817961"
Nov 02 22:36:08.940378 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:08.940339    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:36:14.820067 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:14.820030    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:36:18.893964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.893923    4929 generic.go:356] "Generic (PLEG): container finished" podID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerID="d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287" exitCode=0
Nov 02 22:36:18.894421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.893994    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerDied","Data":"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"}
Nov 02 22:36:18.894421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.894394    4929 scope.go:122] "RemoveContainer" containerID="d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"
Nov 02 22:36:18.895673 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.895641    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:18.895784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.895717    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:18.895784 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:18.895750    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:36:19.786035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:19.785988    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:19.816983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:19.816948    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:36:19.817167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:19.817031    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:36:19.904854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:19.904820    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerStarted","Data":"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"}
Nov 02 22:36:20.627447 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:20.627406    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:20.629351 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:20.629319    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 22:36:20.629505 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:20.629381    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 22:36:21.627710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:21.627665    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 22:36:21.628241 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:21.627726    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 22:36:22.583730 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.583685    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:36:22.587684 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.587642    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.596365 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.596326    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-93446a\"/\"openshift-service-ca.crt\""
Nov 02 22:36:22.597177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.597151    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-93446a\"/\"simulator-single-main-93446a-kserve-self-signed-certs\""
Nov 02 22:36:22.597349 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.597204    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-93446a\"/\"default-dockercfg-hgs6c\""
Nov 02 22:36:22.599675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.597273    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-93446a\"/\"kube-root-ca.crt\""
Nov 02 22:36:22.600379 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.600358    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:36:22.627494 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.627456    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 22:36:22.627723 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.627547    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 22:36:22.710526 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710478    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.711031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710634    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.711031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710678    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.711031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710695    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.711031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710854    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.711031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.710889    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsz7t\" (UniqueName: \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812190    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812421 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812245    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsz7t\" (UniqueName: \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812427    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812513    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812580 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812558    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812583    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812815    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.812984 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.812869    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.813031 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.813016    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.814785 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.814760    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.814943 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.814925    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.820431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.820407    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsz7t\" (UniqueName: \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\") pod \"simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") " pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:22.901904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:22.901864    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:23.204115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:23.204085    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:36:23.206373 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:36:23.206342    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05cbb00a_a4c6_4b65_853a_7ff20c2a6912.slice/crio-0d85a77dcebb7a8ffce810840b750e56a6df01d9e7bfeaaf945aafdfb4538169 WatchSource:0}: Error finding container 0d85a77dcebb7a8ffce810840b750e56a6df01d9e7bfeaaf945aafdfb4538169: Status 404 returned error can't find the container with id 0d85a77dcebb7a8ffce810840b750e56a6df01d9e7bfeaaf945aafdfb4538169
Nov 02 22:36:23.627584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:23.627521    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 22:36:23.627773 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:23.627604    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 22:36:23.925685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:23.925590    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" event={"ID":"05cbb00a-a4c6-4b65-853a-7ff20c2a6912","Type":"ContainerStarted","Data":"7f5b2de4652919b56c5f4631fcaf5c585eacf034bbe8185a88d741412fd7f8aa"}
Nov 02 22:36:23.925685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:23.925635    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" event={"ID":"05cbb00a-a4c6-4b65-853a-7ff20c2a6912","Type":"ContainerStarted","Data":"0d85a77dcebb7a8ffce810840b750e56a6df01d9e7bfeaaf945aafdfb4538169"}
Nov 02 22:36:24.253630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:24.250676    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" podStartSLOduration=2.250651107 podStartE2EDuration="2.250651107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:23.959264003 +0000 UTC m=+2321.267354811" watchObservedRunningTime="2026-11-02 22:36:24.250651107 +0000 UTC m=+2321.558741915"
Nov 02 22:36:24.253630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:24.251009    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:36:24.627666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:24.627624    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 22:36:24.627859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:24.627685    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 22:36:24.902983 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:24.902874    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:25.631173 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:25.631133    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:25.631593 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:25.631441    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:25.632277 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:25.632254    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:36:27.515986 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:36:27.515939    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:36:32.902582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:32.902521    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:32.917212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:32.917180    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:32.981637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:32.981606    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:36:40.650225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:36:40.650183    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:36:57.497382 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:36:57.497334    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:37:04.252927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:04.252886    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:37:08.974569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:08.974522    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:37:14.222615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.222575    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:37:14.233069 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.233032    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:37:14.575778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.575653    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:37:14.576004 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.575921    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerName="maas-api" containerID="cri-o://f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc" gracePeriod=30
Nov 02 22:37:14.839598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.839416    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:37:14.959977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.959938    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\") pod \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") "
Nov 02 22:37:14.960163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.960022    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\") pod \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") "
Nov 02 22:37:14.960163 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.960123    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\") pod \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") "
Nov 02 22:37:14.960278 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.960190    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\" (UniqueName: \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\") pod \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\" (UID: \"59933d41-ed50-4532-8f74-ca6ec99d4e0b\") "
Nov 02 22:37:14.960501 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.960476    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca" pod "59933d41-ed50-4532-8f74-ca6ec99d4e0b" (UID: "59933d41-ed50-4532-8f74-ca6ec99d4e0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:37:14.960662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.960638    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59933d41-ed50-4532-8f74-ca6ec99d4e0b-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:14.962326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.962299    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls" pod "59933d41-ed50-4532-8f74-ca6ec99d4e0b" (UID: "59933d41-ed50-4532-8f74-ca6ec99d4e0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:37:14.962439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.962321    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht" pod "59933d41-ed50-4532-8f74-ca6ec99d4e0b" (UID: "59933d41-ed50-4532-8f74-ca6ec99d4e0b"). InnerVolumeSpecName "kube-api-access-dk2ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:14.962439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:14.962412    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs" pod "59933d41-ed50-4532-8f74-ca6ec99d4e0b" (UID: "59933d41-ed50-4532-8f74-ca6ec99d4e0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:37:15.062068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.062031    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:15.062068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.062063    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59933d41-ed50-4532-8f74-ca6ec99d4e0b-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:15.062068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.062075    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2ht\" (UniqueName: \"kubernetes.io/projected/59933d41-ed50-4532-8f74-ca6ec99d4e0b-kube-api-access-dk2ht\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:15.195490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.195449    4929 generic.go:356] "Generic (PLEG): container finished" podID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerID="f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc" exitCode=0
Nov 02 22:37:15.195490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.195484    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" event={"ID":"59933d41-ed50-4532-8f74-ca6ec99d4e0b","Type":"ContainerDied","Data":"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"}
Nov 02 22:37:15.195750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.195507    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"
Nov 02 22:37:15.195750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.195558    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" event={"ID":"59933d41-ed50-4532-8f74-ca6ec99d4e0b","Type":"ContainerDied","Data":"b25f9e005279fb9ab47a9482e9ee65004d86a3574eed68e7b00cad9833184efc"}
Nov 02 22:37:15.195750 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.195587    4929 scope.go:122] "RemoveContainer" containerID="f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"
Nov 02 22:37:15.205726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.205700    4929 scope.go:122] "RemoveContainer" containerID="f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"
Nov 02 22:37:15.206066 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:37:15.206046    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc\": container with ID starting with f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc not found: ID does not exist" containerID="f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"
Nov 02 22:37:15.206143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.206080    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc"} err="failed to get container status \"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc\": rpc error: code = NotFound desc = could not find container \"f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc\": container with ID starting with f532aeb8fa1be4dffd42bb38328d8aebe52a66a770b78bd9302b8d41287b45fc not found: ID does not exist"
Nov 02 22:37:15.218454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.218420    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:37:15.224164 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:15.224138    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb"]
Nov 02 22:37:16.266657 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:16.266619    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:37:16.802938 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:16.802903    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" path="/var/lib/kubelet/pods/59933d41-ed50-4532-8f74-ca6ec99d4e0b/volumes"
Nov 02 22:37:17.815299 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:17.815247    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-93446a-b55f5b575-clfqb" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.167:8443/health\": context deadline exceeded"
Nov 02 22:37:19.223354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:19.223315    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing" containerID="cri-o://d3fc2a69ece4d866b565f7ed19f349af2fae5961f8f1b1df4ac0333b22009d88" gracePeriod=25
Nov 02 22:37:19.233968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:19.233915    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing" containerID="cri-o://21ad4047c0166012443007dc4cc3fbcf26446597f56009fde5a042ef90d71aea" gracePeriod=25
Nov 02 22:37:19.815380 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:19.815339    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:37:19.815603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:19.815343    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:37:21.401564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:21.401510    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:37:21.402065 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:21.401773    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" containerID="cri-o://5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe" gracePeriod=30
Nov 02 22:37:23.995074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:23.995024    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" podUID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" containerName="main" containerID="cri-o://7f5b2de4652919b56c5f4631fcaf5c585eacf034bbe8185a88d741412fd7f8aa" gracePeriod=45
Nov 02 22:37:24.241797 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.241759    4929 generic.go:356] "Generic (PLEG): container finished" podID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" containerID="7f5b2de4652919b56c5f4631fcaf5c585eacf034bbe8185a88d741412fd7f8aa" exitCode=0
Nov 02 22:37:24.241954 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.241826    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" event={"ID":"05cbb00a-a4c6-4b65-853a-7ff20c2a6912","Type":"ContainerDied","Data":"7f5b2de4652919b56c5f4631fcaf5c585eacf034bbe8185a88d741412fd7f8aa"}
Nov 02 22:37:24.259851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.259754    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:37:24.357802 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357765    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357829    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357848    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357867    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\" (UniqueName: \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357944    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.357983    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\" (UniqueName: \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\") pod \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\" (UID: \"05cbb00a-a4c6-4b65-853a-7ff20c2a6912\") "
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358110    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358161    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358269    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358381    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358412    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:24.358482 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.358428    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:24.360423 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.360394    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:24.360527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.360452    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "kube-api-access-dsz7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:24.360527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.360476    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs" pod "05cbb00a-a4c6-4b65-853a-7ff20c2a6912" (UID: "05cbb00a-a4c6-4b65-853a-7ff20c2a6912"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:37:24.459236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.459185    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:24.459236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.459227    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:24.459236 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:24.459238    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsz7t\" (UniqueName: \"kubernetes.io/projected/05cbb00a-a4c6-4b65-853a-7ff20c2a6912-kube-api-access-dsz7t\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:25.248177 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.248147    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"
Nov 02 22:37:25.248666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.248139    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc" event={"ID":"05cbb00a-a4c6-4b65-853a-7ff20c2a6912","Type":"ContainerDied","Data":"0d85a77dcebb7a8ffce810840b750e56a6df01d9e7bfeaaf945aafdfb4538169"}
Nov 02 22:37:25.248666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.248271    4929 scope.go:122] "RemoveContainer" containerID="7f5b2de4652919b56c5f4631fcaf5c585eacf034bbe8185a88d741412fd7f8aa"
Nov 02 22:37:25.263889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.263854    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:37:25.269019 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.268986    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-93446a/simulator-single-main-93446a-kserve-9dd8ff5f5-qgclc"]
Nov 02 22:37:25.348630 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.348591    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:37:25.349047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349028    4929 cpu_manager.go:414] "containerMap: removing container" podUID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" containerName="main"
Nov 02 22:37:25.349047 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349045    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" containerName="main"
Nov 02 22:37:25.349260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349071    4929 cpu_manager.go:414] "containerMap: removing container" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerName="maas-api"
Nov 02 22:37:25.349260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349078    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerName="maas-api"
Nov 02 22:37:25.349260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349199    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="59933d41-ed50-4532-8f74-ca6ec99d4e0b" containerName="maas-api"
Nov 02 22:37:25.349260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.349222    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" containerName="main"
Nov 02 22:37:25.353948 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.353922    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.362969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.362408    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:37:25.362969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.362633    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-abe63d40-kserve-self-signed-certs\""
Nov 02 22:37:25.368705 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368670    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.368900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368723    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pbd\" (UniqueName: \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.368900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368813    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.368900 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368860    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.369039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368949    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.369039 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.368985    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.469841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469798    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.469841 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469843    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pbd\" (UniqueName: \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469879    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469916    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469959    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.469978    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.470288    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.470372    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.470437 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.470406    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.472677 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.472651    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.473027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.473003    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.478934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.478890    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pbd\" (UniqueName: \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\") pod \"e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") " pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.634788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.634744    4929 patch_prober.go:29] interesting pod/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:37:25.634964 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.634817    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:37:25.668119 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.668072    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:25.943906 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:25.943876    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:37:25.946230 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:37:25.946203    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod432d7c56_ce17_4071_b3e5_b0e7b6a7ea30.slice/crio-9a2dff85798f9b5e8fcda534025a51b974caebaf9d91560ec5481d2912ecf47a WatchSource:0}: Error finding container 9a2dff85798f9b5e8fcda534025a51b974caebaf9d91560ec5481d2912ecf47a: Status 404 returned error can't find the container with id 9a2dff85798f9b5e8fcda534025a51b974caebaf9d91560ec5481d2912ecf47a
Nov 02 22:37:26.257414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.257316    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" event={"ID":"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30","Type":"ContainerStarted","Data":"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"}
Nov 02 22:37:26.257414 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.257357    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" event={"ID":"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30","Type":"ContainerStarted","Data":"9a2dff85798f9b5e8fcda534025a51b974caebaf9d91560ec5481d2912ecf47a"}
Nov 02 22:37:26.273725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.273672    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" podStartSLOduration=1.27365899 podStartE2EDuration="1.27365899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:26.272211649 +0000 UTC m=+2383.580302454" watchObservedRunningTime="2026-11-02 22:37:26.27365899 +0000 UTC m=+2383.581749792"
Nov 02 22:37:26.661924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.661896    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:37:26.668372 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.668342    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:26.680834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680746    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.680834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680796    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680852    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680884    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680918    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.680952    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681109 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681024    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681130    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.681319 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681166    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\") pod \"a1d480a9-7703-447f-9863-6e877ba1c7f4\" (UID: \"a1d480a9-7703-447f-9863-6e877ba1c7f4\") "
Nov 02 22:37:26.682431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681608    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:26.682431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681794    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:37:26.682431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.681840    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:26.682431 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.682066    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:26.682772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.682636    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:26.683799 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.683771    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:26.684405 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.684369    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "kube-api-access-nsdfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:26.684581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.684446    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:37:26.685310 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.685285    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy" pod "a1d480a9-7703-447f-9863-6e877ba1c7f4" (UID: "a1d480a9-7703-447f-9863-6e877ba1c7f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782513    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d480a9-7703-447f-9863-6e877ba1c7f4-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782583    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782597    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782608    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782618    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782627    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdfc\" (UniqueName: \"kubernetes.io/projected/a1d480a9-7703-447f-9863-6e877ba1c7f4-kube-api-access-nsdfc\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782639    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782647    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.782996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.782655    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d480a9-7703-447f-9863-6e877ba1c7f4-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:26.803192 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.803153    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cbb00a-a4c6-4b65-853a-7ff20c2a6912" path="/var/lib/kubelet/pods/05cbb00a-a4c6-4b65-853a-7ff20c2a6912/volumes"
Nov 02 22:37:26.846737 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:26.846697    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:37:27.264525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.264486    4929 generic.go:356] "Generic (PLEG): container finished" podID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerID="5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe" exitCode=0
Nov 02 22:37:27.265049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.264569    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"
Nov 02 22:37:27.265049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.264572    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerDied","Data":"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"}
Nov 02 22:37:27.265049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.264637    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw" event={"ID":"a1d480a9-7703-447f-9863-6e877ba1c7f4","Type":"ContainerDied","Data":"515fcf23d27fb84ab11fc26be94a2d185c6812f761d67d0abdab22a28455120a"}
Nov 02 22:37:27.265049 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.264662    4929 scope.go:122] "RemoveContainer" containerID="5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"
Nov 02 22:37:27.273823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.273801    4929 scope.go:122] "RemoveContainer" containerID="d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"
Nov 02 22:37:27.282184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.282150    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:37:27.282703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.282684    4929 scope.go:122] "RemoveContainer" containerID="5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"
Nov 02 22:37:27.283031 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:37:27.283005    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe\": container with ID starting with 5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe not found: ID does not exist" containerID="5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"
Nov 02 22:37:27.283112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.283045    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe"} err="failed to get container status \"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe\": rpc error: code = NotFound desc = could not find container \"5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe\": container with ID starting with 5402e3840fa54558245b0c0fd184a23cf339703df19b40271adc5d8ea43de7fe not found: ID does not exist"
Nov 02 22:37:27.283112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.283069    4929 scope.go:122] "RemoveContainer" containerID="d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"
Nov 02 22:37:27.283528 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:37:27.283474    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287\": container with ID starting with d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287 not found: ID does not exist" containerID="d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"
Nov 02 22:37:27.283528 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.283513    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287"} err="failed to get container status \"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287\": rpc error: code = NotFound desc = could not find container \"d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287\": container with ID starting with d6a4dd1670ffa8cdf618bdb82bccb2321d52d2e4867b6272b79d2bdcc6005287 not found: ID does not exist"
Nov 02 22:37:27.285665 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:27.285641    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93446a-openshift-default-6d9c47bf9c9x9sw"]
Nov 02 22:37:27.510802 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:37:27.510749    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:37:28.824559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:28.824358    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" path="/var/lib/kubelet/pods/a1d480a9-7703-447f-9863-6e877ba1c7f4/volumes"
Nov 02 22:37:29.839560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:29.838680    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:37:29.852566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:29.843651    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:37:35.669095 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:35.669045    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:35.682561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:35.682500    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:36.360342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:36.360306    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:37:39.366486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.366437    4929 generic.go:356] "Generic (PLEG): container finished" podID="592e6358-00fc-4e55-88b5-746a918e54d2" containerID="d3fc2a69ece4d866b565f7ed19f349af2fae5961f8f1b1df4ac0333b22009d88" exitCode=0
Nov 02 22:37:39.367079 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.366514    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" event={"ID":"592e6358-00fc-4e55-88b5-746a918e54d2","Type":"ContainerDied","Data":"d3fc2a69ece4d866b565f7ed19f349af2fae5961f8f1b1df4ac0333b22009d88"}
Nov 02 22:37:39.368188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.368160    4929 generic.go:356] "Generic (PLEG): container finished" podID="8e15a015-048e-4711-b7bc-a955814f2662" containerID="21ad4047c0166012443007dc4cc3fbcf26446597f56009fde5a042ef90d71aea" exitCode=0
Nov 02 22:37:39.368357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.368233    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" event={"ID":"8e15a015-048e-4711-b7bc-a955814f2662","Type":"ContainerDied","Data":"21ad4047c0166012443007dc4cc3fbcf26446597f56009fde5a042ef90d71aea"}
Nov 02 22:37:39.487666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.487599    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:37:39.520516 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.520489    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:37:39.555357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.555318    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\") pod \"8e15a015-048e-4711-b7bc-a955814f2662\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") "
Nov 02 22:37:39.555556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.555443    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\" (UniqueName: \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\") pod \"592e6358-00fc-4e55-88b5-746a918e54d2\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") "
Nov 02 22:37:39.555556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.555488    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\" (UniqueName: \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\") pod \"8e15a015-048e-4711-b7bc-a955814f2662\" (UID: \"8e15a015-048e-4711-b7bc-a955814f2662\") "
Nov 02 22:37:39.555556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.555515    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\") pod \"592e6358-00fc-4e55-88b5-746a918e54d2\" (UID: \"592e6358-00fc-4e55-88b5-746a918e54d2\") "
Nov 02 22:37:39.555945 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.555793    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume" pod "8e15a015-048e-4711-b7bc-a955814f2662" (UID: "8e15a015-048e-4711-b7bc-a955814f2662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:37:39.556078 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.556006    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume" pod "592e6358-00fc-4e55-88b5-746a918e54d2" (UID: "592e6358-00fc-4e55-88b5-746a918e54d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:37:39.557735 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.557712    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg" pod "592e6358-00fc-4e55-88b5-746a918e54d2" (UID: "592e6358-00fc-4e55-88b5-746a918e54d2"). InnerVolumeSpecName "kube-api-access-47kjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:39.557846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.557752    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g" pod "8e15a015-048e-4711-b7bc-a955814f2662" (UID: "8e15a015-048e-4711-b7bc-a955814f2662"). InnerVolumeSpecName "kube-api-access-r9r7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:37:39.657201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.657153    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47kjg\" (UniqueName: \"kubernetes.io/projected/592e6358-00fc-4e55-88b5-746a918e54d2-kube-api-access-47kjg\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:39.657201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.657190    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9r7g\" (UniqueName: \"kubernetes.io/projected/8e15a015-048e-4711-b7bc-a955814f2662-kube-api-access-r9r7g\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:39.657201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.657200    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592e6358-00fc-4e55-88b5-746a918e54d2-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:39.657201 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:39.657210    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e15a015-048e-4711-b7bc-a955814f2662-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:37:40.373588 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.373529    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"
Nov 02 22:37:40.374066 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.373553    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj" event={"ID":"592e6358-00fc-4e55-88b5-746a918e54d2","Type":"ContainerDied","Data":"d14853f251344a50ca3fc72fbf9358e1f4fcfc8df83d2965af74da3f679ecd8f"}
Nov 02 22:37:40.374066 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.373675    4929 scope.go:122] "RemoveContainer" containerID="d3fc2a69ece4d866b565f7ed19f349af2fae5961f8f1b1df4ac0333b22009d88"
Nov 02 22:37:40.375093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.374990    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"
Nov 02 22:37:40.375093 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.375016    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct" event={"ID":"8e15a015-048e-4711-b7bc-a955814f2662","Type":"ContainerDied","Data":"74c5f2f2f176cf79c7eaa578ad141a5dcd4122c51b97e05347e18e2f0bb51933"}
Nov 02 22:37:40.383206 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.383180    4929 scope.go:122] "RemoveContainer" containerID="21ad4047c0166012443007dc4cc3fbcf26446597f56009fde5a042ef90d71aea"
Nov 02 22:37:40.399074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.399046    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:37:40.404250 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.404221    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93446a-564cbjs4ct"]
Nov 02 22:37:40.413478 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.413447    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:37:40.417768 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.417744    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93446a-794d4f4c58qzkj"]
Nov 02 22:37:40.802935 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.802902    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" path="/var/lib/kubelet/pods/592e6358-00fc-4e55-88b5-746a918e54d2/volumes"
Nov 02 22:37:40.803312 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:40.803299    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e15a015-048e-4711-b7bc-a955814f2662" path="/var/lib/kubelet/pods/8e15a015-048e-4711-b7bc-a955814f2662/volumes"
Nov 02 22:37:42.975615 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.975569    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:37:42.976184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976163    4929 cpu_manager.go:414] "containerMap: removing container" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976187    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976222    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976231    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976254    4929 cpu_manager.go:414] "containerMap: removing container" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976262    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:37:42.976276 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976272    4929 cpu_manager.go:414] "containerMap: removing container" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing"
Nov 02 22:37:42.976886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976281    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing"
Nov 02 22:37:42.976886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976393    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:37:42.976886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976408    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e15a015-048e-4711-b7bc-a955814f2662" containerName="payload-pre-processing"
Nov 02 22:37:42.976886 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.976423    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="592e6358-00fc-4e55-88b5-746a918e54d2" containerName="payload-processing"
Nov 02 22:37:42.981364 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.981339    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:42.987975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.987940    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-abe63d40-kserve-self-signed-certs\""
Nov 02 22:37:42.990296 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:42.990259    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:37:43.094167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094123    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.094167 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094169    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.094407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094308    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.094407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094354    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89jf\" (UniqueName: \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.094407 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094390    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.094515 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.094424    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195123    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195177    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195225    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195242    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195280    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195382 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195308    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89jf\" (UniqueName: \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195707 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195667    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195772 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195671    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.195834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.195793    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.197681 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.197655    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.197863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.197844    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.202703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.202678    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89jf\" (UniqueName: \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\") pod \"e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") " pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.298415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.298316    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:43.577042 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:43.576961    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:37:43.579388 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:37:43.579347    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1180118_b466_4355_add5_4aec061603bb.slice/crio-dc82be0b88a8e7148be9decfa86856d413a9220ab727e008b5d96d2fec3072e3 WatchSource:0}: Error finding container dc82be0b88a8e7148be9decfa86856d413a9220ab727e008b5d96d2fec3072e3: Status 404 returned error can't find the container with id dc82be0b88a8e7148be9decfa86856d413a9220ab727e008b5d96d2fec3072e3
Nov 02 22:37:44.398541 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:44.398484    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" event={"ID":"d1180118-b466-4355-add5-4aec061603bb","Type":"ContainerStarted","Data":"8e21e9121ca5ae4f72abed647175902c84daf7f29310cb024c5991c1442a5068"}
Nov 02 22:37:44.398541 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:44.398544    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" event={"ID":"d1180118-b466-4355-add5-4aec061603bb","Type":"ContainerStarted","Data":"dc82be0b88a8e7148be9decfa86856d413a9220ab727e008b5d96d2fec3072e3"}
Nov 02 22:37:44.423413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:44.423346    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" podStartSLOduration=2.423327204 podStartE2EDuration="2.423327204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:44.422958566 +0000 UTC m=+2401.731049371" watchObservedRunningTime="2026-11-02 22:37:44.423327204 +0000 UTC m=+2401.731418004"
Nov 02 22:37:44.577759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:44.577720    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:37:45.298740 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:45.298696    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:45.655334 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:45.655297    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:37:45.658408 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:45.658375    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:37:53.298844 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:53.298800    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:53.311932 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:53.311896    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:53.528977 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:53.528946    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:37:57.496404 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:37:57.496370    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:37:59.646313 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:37:59.646275    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:09.858854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:09.858812    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:38:11.068425 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:11.068387    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:11.199650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:11.199613    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:38:16.395240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:16.395197    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:24.884706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:24.884645    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" podUID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" containerName="main" containerID="cri-o://f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381" gracePeriod=45
Nov 02 22:38:25.145005 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.144929    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:38:25.162692 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162646    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.162882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162733    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.162882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162766    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\" (UniqueName: \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.162882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162791    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162913    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162952    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\" (UniqueName: \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\") pod \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\" (UID: \"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30\") "
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162947    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.162988    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.163144    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.163290    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.163308    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.163672 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.163323    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.167469 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.167439    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "kube-api-access-l2pbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:38:25.167646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.167438    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:38:25.167646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.167470    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm" pod "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" (UID: "432d7c56-ce17-4071-b3e5-b0e7b6a7ea30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:25.264614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.264522    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.264614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.264610    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2pbd\" (UniqueName: \"kubernetes.io/projected/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-kube-api-access-l2pbd\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.264614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.264622    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:25.674584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.674517    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:25.738395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.738359    4929 generic.go:356] "Generic (PLEG): container finished" podID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" containerID="f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381" exitCode=0
Nov 02 22:38:25.738598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.738406    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"
Nov 02 22:38:25.738598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.738441    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" event={"ID":"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30","Type":"ContainerDied","Data":"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"}
Nov 02 22:38:25.738598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.738490    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr" event={"ID":"432d7c56-ce17-4071-b3e5-b0e7b6a7ea30","Type":"ContainerDied","Data":"9a2dff85798f9b5e8fcda534025a51b974caebaf9d91560ec5481d2912ecf47a"}
Nov 02 22:38:25.738598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.738511    4929 scope.go:122] "RemoveContainer" containerID="f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"
Nov 02 22:38:25.748461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.748427    4929 scope.go:122] "RemoveContainer" containerID="f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"
Nov 02 22:38:25.748829 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:38:25.748807    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381\": container with ID starting with f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381 not found: ID does not exist" containerID="f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"
Nov 02 22:38:25.748912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.748839    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381"} err="failed to get container status \"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381\": rpc error: code = NotFound desc = could not find container \"f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381\": container with ID starting with f7ebc5e06c69cc3a9503a738925916505bb8a93cea1af9ec313120b2bca94381 not found: ID does not exist"
Nov 02 22:38:25.759320 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.759291    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:38:25.764061 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:25.764033    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-abe63d40-kserve-8f6f4fc85-7flvr"]
Nov 02 22:38:26.234856 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:26.234794    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" podUID="d1180118-b466-4355-add5-4aec061603bb" containerName="main" containerID="cri-o://8e21e9121ca5ae4f72abed647175902c84daf7f29310cb024c5991c1442a5068" gracePeriod=45
Nov 02 22:38:26.530292 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:26.530180    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" podUID="d1180118-b466-4355-add5-4aec061603bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.172:8000/ready\": dial tcp 10.133.0.172:8000: connect: connection refused"
Nov 02 22:38:26.767253 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:26.767188    4929 generic.go:356] "Generic (PLEG): container finished" podID="d1180118-b466-4355-add5-4aec061603bb" containerID="8e21e9121ca5ae4f72abed647175902c84daf7f29310cb024c5991c1442a5068" exitCode=0
Nov 02 22:38:26.767689 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:26.767553    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" event={"ID":"d1180118-b466-4355-add5-4aec061603bb","Type":"ContainerDied","Data":"8e21e9121ca5ae4f72abed647175902c84daf7f29310cb024c5991c1442a5068"}
Nov 02 22:38:27.055625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.054114    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" path="/var/lib/kubelet/pods/432d7c56-ce17-4071-b3e5-b0e7b6a7ea30/volumes"
Nov 02 22:38:27.345170 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.345124    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.486880    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.486972    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.487012    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.487109    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.487197    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.487624 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.487261    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\" (UniqueName: \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\") pod \"d1180118-b466-4355-add5-4aec061603bb\" (UID: \"d1180118-b466-4355-add5-4aec061603bb\") "
Nov 02 22:38:27.533987 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.524689    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:27.534432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.534395    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:27.540255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.540218    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "kube-api-access-g89jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:38:27.601043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.600927    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89jf\" (UniqueName: \"kubernetes.io/projected/d1180118-b466-4355-add5-4aec061603bb-kube-api-access-g89jf\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.601043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.600975    4929 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-home\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.601043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.600995    4929 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-model-cache\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.601764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.601734    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:27.620113 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.619975    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:38:27.667598 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.667560    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs" pod "d1180118-b466-4355-add5-4aec061603bb" (UID: "d1180118-b466-4355-add5-4aec061603bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:38:27.702461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.702427    4929 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-dshm\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.702753 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.702734    4929 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1180118-b466-4355-add5-4aec061603bb-tmp-dir\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.702910 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.702896    4929 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1180118-b466-4355-add5-4aec061603bb-tls-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:38:27.761639 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:38:27.758190    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:38:27.793577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.791395    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz" event={"ID":"d1180118-b466-4355-add5-4aec061603bb","Type":"ContainerDied","Data":"dc82be0b88a8e7148be9decfa86856d413a9220ab727e008b5d96d2fec3072e3"}
Nov 02 22:38:27.793577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.791460    4929 scope.go:122] "RemoveContainer" containerID="8e21e9121ca5ae4f72abed647175902c84daf7f29310cb024c5991c1442a5068"
Nov 02 22:38:27.793577 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.791624    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"
Nov 02 22:38:27.858456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.858382    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:38:27.862076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:27.862040    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-abe63d40-kserve-5f7bd47945-bgflz"]
Nov 02 22:38:28.806038 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:28.805998    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1180118-b466-4355-add5-4aec061603bb" path="/var/lib/kubelet/pods/d1180118-b466-4355-add5-4aec061603bb/volumes"
Nov 02 22:38:29.866568 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:29.865548    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:45.936670 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:45.936630    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:38:47.331088 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.331040    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"]
Nov 02 22:38:47.331859 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.331831    4929 cpu_manager.go:414] "containerMap: removing container" podUID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" containerName="main"
Nov 02 22:38:47.331912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.331864    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" containerName="main"
Nov 02 22:38:47.331912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.331882    4929 cpu_manager.go:414] "containerMap: removing container" podUID="d1180118-b466-4355-add5-4aec061603bb" containerName="main"
Nov 02 22:38:47.331912 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.331890    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1180118-b466-4355-add5-4aec061603bb" containerName="main"
Nov 02 22:38:47.332074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.332055    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="432d7c56-ce17-4071-b3e5-b0e7b6a7ea30" containerName="main"
Nov 02 22:38:47.332115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.332080    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1180118-b466-4355-add5-4aec061603bb" containerName="main"
Nov 02 22:38:47.332115 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.332091    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d480a9-7703-447f-9863-6e877ba1c7f4" containerName="istio-proxy"
Nov 02 22:38:47.336071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.336044    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.343919 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.343886    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"]
Nov 02 22:38:47.355326 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.355287    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7672b2f9-0101-4ad1-90ae-c73989202d13-openshift-service-ca\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.355581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.355354    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xnr\" (UniqueName: \"kubernetes.io/projected/7672b2f9-0101-4ad1-90ae-c73989202d13-kube-api-access-j2xnr\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.355581 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.355394    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-maas-api-tls\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.355727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.355605    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-metrics-certs\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.456671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.456631    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7672b2f9-0101-4ad1-90ae-c73989202d13-openshift-service-ca\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.456671 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.456680    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xnr\" (UniqueName: \"kubernetes.io/projected/7672b2f9-0101-4ad1-90ae-c73989202d13-kube-api-access-j2xnr\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.456926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.456701    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-maas-api-tls\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.456926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.456759    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-metrics-certs\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.457376 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.457349    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7672b2f9-0101-4ad1-90ae-c73989202d13-openshift-service-ca\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.459467 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.459441    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-metrics-certs\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.459636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.459483    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7672b2f9-0101-4ad1-90ae-c73989202d13-maas-api-tls\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.464452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.464423    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xnr\" (UniqueName: \"kubernetes.io/projected/7672b2f9-0101-4ad1-90ae-c73989202d13-kube-api-access-j2xnr\") pod \"maas-api-7b994c7666-hr4rf\" (UID: \"7672b2f9-0101-4ad1-90ae-c73989202d13\") " pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.654420 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.654382    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:47.936845 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.936809    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"]
Nov 02 22:38:47.939035 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:38:47.939004    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7672b2f9_0101_4ad1_90ae_c73989202d13.slice/crio-e8e14fc85b88cef708711ce75cbe7e47ba418116754072079e68b1bc669a526e WatchSource:0}: Error finding container e8e14fc85b88cef708711ce75cbe7e47ba418116754072079e68b1bc669a526e: Status 404 returned error can't find the container with id e8e14fc85b88cef708711ce75cbe7e47ba418116754072079e68b1bc669a526e
Nov 02 22:38:47.940429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:47.940404    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:38:48.914033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:48.913992    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf" event={"ID":"7672b2f9-0101-4ad1-90ae-c73989202d13","Type":"ContainerStarted","Data":"cbfd0eb5bc49dadadf840b928d000b23702496f8324087438a3b95f69d245b2f"}
Nov 02 22:38:48.914454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:48.914056    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf" event={"ID":"7672b2f9-0101-4ad1-90ae-c73989202d13","Type":"ContainerStarted","Data":"e8e14fc85b88cef708711ce75cbe7e47ba418116754072079e68b1bc669a526e"}
Nov 02 22:38:48.914454 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:48.914214    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:48.930662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:48.930609    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf" podStartSLOduration=1.516011497 podStartE2EDuration="1.930594089s" totalImagesPullingTime="414.582592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:48.929704264 +0000 UTC m=+2466.237795070" watchObservedRunningTime="2026-11-02 22:38:48.930594089 +0000 UTC m=+2466.238684894"
Nov 02 22:38:54.492344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.492251    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:38:54.496027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.496003    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.499771 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.499737    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-555ba599-openshift-default-dockercfg-9zj26\""
Nov 02 22:38:54.507233 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.507196    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:38:54.522275 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522233    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522307    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522462 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522388    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522476    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjh2s\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522519    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522589 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522564    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522621    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522713    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.522809 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.522745    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624011 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.623976    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624039    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624067    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624120    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjh2s\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624162    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624235 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624192    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624278    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624322    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.624502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.624360    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.625396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.625361    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.625907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.625873    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.626112 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.626089    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.626214 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.626137    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.626332 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.626309    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.627879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.627856    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.628486 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.628458    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.633215 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.633185    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.633453 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.633426    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjh2s\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\") pod \"e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") " pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.817357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.816755    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:54.925383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:54.925339    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b994c7666-hr4rf"
Nov 02 22:38:55.097488 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.097381    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:38:55.100050 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:38:55.100020    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30454237_881c_4c13_979e_1bbfb8deb44a.slice/crio-5632fd34d480aff785990bf9dbf54491fcb0ca310e138c303cb9353f018d8df6 WatchSource:0}: Error finding container 5632fd34d480aff785990bf9dbf54491fcb0ca310e138c303cb9353f018d8df6: Status 404 returned error can't find the container with id 5632fd34d480aff785990bf9dbf54491fcb0ca310e138c303cb9353f018d8df6
Nov 02 22:38:55.102404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.102367    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:38:55.102513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.102441    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:38:55.102513 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.102475    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:38:55.954298 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.954254    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerStarted","Data":"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"}
Nov 02 22:38:55.954763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.954312    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerStarted","Data":"5632fd34d480aff785990bf9dbf54491fcb0ca310e138c303cb9353f018d8df6"}
Nov 02 22:38:55.973391 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:55.973333    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podStartSLOduration=1.973315706 podStartE2EDuration="1.973315706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:55.97238206 +0000 UTC m=+2473.280472876" watchObservedRunningTime="2026-11-02 22:38:55.973315706 +0000 UTC m=+2473.281406511"
Nov 02 22:38:56.817055 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:56.817015    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:56.822609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:56.822581    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:56.959265 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:56.959235    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:56.960187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:56.960158    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:38:57.496614 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:38:57.496578    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:38:58.627663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.627592    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:38:58.631613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.631582    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.644108 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.644075    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-555ba599-serving-cert\""
Nov 02 22:38:58.648646 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.645793    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:38:58.669009 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.668966    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfnl\" (UniqueName: \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.669244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.669109    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.669244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.669142    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.669244 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.669170    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.758579 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.758517    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:38:58.762124 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.762104    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.769706 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769677    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-555ba599\""
Nov 02 22:38:58.769850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769821    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.769850 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769843    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-555ba599-dockercfg-2q4c5\""
Nov 02 22:38:58.769985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769867    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.769985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769907    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.769985 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.769975    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfnl\" (UniqueName: \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.770934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.770905    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.771569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.771524    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:38:58.773073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.773048    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.773959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.773938    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.782641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.782607    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfnl\" (UniqueName: \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\") pod \"maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.803395 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.803362    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:38:58.806969 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.806928    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.813063 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.813036    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:38:58.871018 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.870973    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrgf\" (UniqueName: \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.871216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.871123    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.871216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.871170    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.871341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.871310    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zb9\" (UniqueName: \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.953123 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.953024    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:58.972685 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.972654    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zb9\" (UniqueName: \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.972846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.972698    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrgf\" (UniqueName: \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.972846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.972757    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.972846 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.972782    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.973480 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.973456    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:58.973611 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.973488    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.981764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.981727    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrgf\" (UniqueName: \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\") pod \"payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:58.981903 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:58.981779    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zb9\" (UniqueName: \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\") pod \"payload-processing-e2e-ait-555ba599-69995448cc-4cr5q\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:59.078960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.078921    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:59.121922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.121885    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:59.238596 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.238558    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:38:59.240984 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:38:59.240950    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e8b8b5_8596_43d3_90b2_8db8f8e72c9d.slice/crio-9096d9de01b1a38dabbd03afd60f31d9d09ecd8a95195e0de50a8ba9c056dace WatchSource:0}: Error finding container 9096d9de01b1a38dabbd03afd60f31d9d09ecd8a95195e0de50a8ba9c056dace: Status 404 returned error can't find the container with id 9096d9de01b1a38dabbd03afd60f31d9d09ecd8a95195e0de50a8ba9c056dace
Nov 02 22:38:59.354342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.354310    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:38:59.356462 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:38:59.356439    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aafb070_c5b1_411c_ba1b_113832444f69.slice/crio-e2570686d6927801805e0f21a8d1cd2c88e5654733c446ecae70d084daeba9a7 WatchSource:0}: Error finding container e2570686d6927801805e0f21a8d1cd2c88e5654733c446ecae70d084daeba9a7: Status 404 returned error can't find the container with id e2570686d6927801805e0f21a8d1cd2c88e5654733c446ecae70d084daeba9a7
Nov 02 22:38:59.406498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.406471    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:38:59.408826 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:38:59.408795    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c15709_2394_447c_bf66_2fbb99b5b5c2.slice/crio-72c4e4c3450331da457501b9c2e49588cdf4a44be68e15a96637a58c47dc4d6f WatchSource:0}: Error finding container 72c4e4c3450331da457501b9c2e49588cdf4a44be68e15a96637a58c47dc4d6f: Status 404 returned error can't find the container with id 72c4e4c3450331da457501b9c2e49588cdf4a44be68e15a96637a58c47dc4d6f
Nov 02 22:38:59.975616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.975500    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" event={"ID":"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d","Type":"ContainerStarted","Data":"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"}
Nov 02 22:38:59.975616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.975588    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" event={"ID":"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d","Type":"ContainerStarted","Data":"9096d9de01b1a38dabbd03afd60f31d9d09ecd8a95195e0de50a8ba9c056dace"}
Nov 02 22:38:59.976111 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.975621    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:38:59.977085 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.977055    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" event={"ID":"1aafb070-c5b1-411c-ba1b-113832444f69","Type":"ContainerStarted","Data":"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"}
Nov 02 22:38:59.977216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.977110    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:38:59.977216 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.977136    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" event={"ID":"1aafb070-c5b1-411c-ba1b-113832444f69","Type":"ContainerStarted","Data":"e2570686d6927801805e0f21a8d1cd2c88e5654733c446ecae70d084daeba9a7"}
Nov 02 22:38:59.978506 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.978484    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" event={"ID":"33c15709-2394-447c-bf66-2fbb99b5b5c2","Type":"ContainerStarted","Data":"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"}
Nov 02 22:38:59.978662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.978523    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" event={"ID":"33c15709-2394-447c-bf66-2fbb99b5b5c2","Type":"ContainerStarted","Data":"72c4e4c3450331da457501b9c2e49588cdf4a44be68e15a96637a58c47dc4d6f"}
Nov 02 22:38:59.978662 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.978576    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:38:59.993973 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:38:59.993899    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" podStartSLOduration=1.581998631 podStartE2EDuration="1.993887196s" totalImagesPullingTime="411.888565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:59.99247908 +0000 UTC m=+2477.300569886" watchObservedRunningTime="2026-11-02 22:38:59.993887196 +0000 UTC m=+2477.301977998"
Nov 02 22:39:00.009338 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:00.009294    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" podStartSLOduration=2.00928236 podStartE2EDuration="2.00928236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:00.008046703 +0000 UTC m=+2477.316137508" watchObservedRunningTime="2026-11-02 22:39:00.00928236 +0000 UTC m=+2477.317373158"
Nov 02 22:39:00.024786 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:00.024717    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" podStartSLOduration=2.024698588 podStartE2EDuration="2.024698588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:00.023943743 +0000 UTC m=+2477.332034550" watchObservedRunningTime="2026-11-02 22:39:00.024698588 +0000 UTC m=+2477.332789395"
Nov 02 22:39:01.041465 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:01.041426    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:39:03.185187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:03.185145    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:39:03.185640 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:03.185421    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" podUID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" containerName="maas-api" containerID="cri-o://6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9" gracePeriod=30
Nov 02 22:39:04.488639 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.488612    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:39:04.633352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633317    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\") pod \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") "
Nov 02 22:39:04.633565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633359    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\") pod \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") "
Nov 02 22:39:04.633565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633452    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\" (UniqueName: \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\") pod \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") "
Nov 02 22:39:04.633709 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633579    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\") pod \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\" (UID: \"ac17491a-49fb-42c8-873d-c4d4a457a2fd\") "
Nov 02 22:39:04.633863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633838    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca" pod "ac17491a-49fb-42c8-873d-c4d4a457a2fd" (UID: "ac17491a-49fb-42c8-873d-c4d4a457a2fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:04.633957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.633941    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac17491a-49fb-42c8-873d-c4d4a457a2fd-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:04.660461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.660407    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls" pod "ac17491a-49fb-42c8-873d-c4d4a457a2fd" (UID: "ac17491a-49fb-42c8-873d-c4d4a457a2fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:39:04.661043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.661014    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4" pod "ac17491a-49fb-42c8-873d-c4d4a457a2fd" (UID: "ac17491a-49fb-42c8-873d-c4d4a457a2fd"). InnerVolumeSpecName "kube-api-access-5s6r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:04.661487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.661420    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs" pod "ac17491a-49fb-42c8-873d-c4d4a457a2fd" (UID: "ac17491a-49fb-42c8-873d-c4d4a457a2fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:39:04.755560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.735196    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:04.755560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.735240    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac17491a-49fb-42c8-873d-c4d4a457a2fd-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:04.755560 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:04.735257    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s6r4\" (UniqueName: \"kubernetes.io/projected/ac17491a-49fb-42c8-873d-c4d4a457a2fd-kube-api-access-5s6r4\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:05.003518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.003427    4929 generic.go:356] "Generic (PLEG): container finished" podID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" containerID="6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9" exitCode=0
Nov 02 22:39:05.003518 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.003472    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"
Nov 02 22:39:05.003749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.003513    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" event={"ID":"ac17491a-49fb-42c8-873d-c4d4a457a2fd","Type":"ContainerDied","Data":"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"}
Nov 02 22:39:05.003749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.003587    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t" event={"ID":"ac17491a-49fb-42c8-873d-c4d4a457a2fd","Type":"ContainerDied","Data":"70c5ab3ad5c314652886b48cceb03cb6da910e3ca826720d5e63df33047d873d"}
Nov 02 22:39:05.003749 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.003616    4929 scope.go:122] "RemoveContainer" containerID="6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"
Nov 02 22:39:05.012763 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.012731    4929 scope.go:122] "RemoveContainer" containerID="6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"
Nov 02 22:39:05.013015 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:05.012992    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9\": container with ID starting with 6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9 not found: ID does not exist" containerID="6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"
Nov 02 22:39:05.013074 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.013027    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9"} err="failed to get container status \"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9\": rpc error: code = NotFound desc = could not find container \"6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9\": container with ID starting with 6ddb63c4df740d422876e87789b8b76edef962ca91f766f33db8909c9f2ec3e9 not found: ID does not exist"
Nov 02 22:39:05.021388 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.021366    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:39:05.025056 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.025033    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-x9q9t"]
Nov 02 22:39:05.989569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:05.989516    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:39:06.803543 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:06.803490    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" path="/var/lib/kubelet/pods/ac17491a-49fb-42c8-873d-c4d4a457a2fd/volumes"
Nov 02 22:39:10.987308 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:10.987274    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:39:10.987891 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:10.987352    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:39:11.042967 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.042880    4929 generic.go:356] "Generic (PLEG): container finished" podID="30454237-881c-4c13-979e-1bbfb8deb44a" containerID="decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d" exitCode=0
Nov 02 22:39:11.043230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.042923    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerDied","Data":"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"}
Nov 02 22:39:11.043848 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.043828    4929 scope.go:122] "RemoveContainer" containerID="decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"
Nov 02 22:39:11.046429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.046395    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:11.046569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.046476    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:11.046569 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.046510    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:11.975318 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:11.975270    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:12.071863 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:12.071817    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerStarted","Data":"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"}
Nov 02 22:39:12.817364 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:12.817317    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:12.818843 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:12.818808    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 22:39:12.818959 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:12.818875    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 22:39:13.817614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:13.817568    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 22:39:13.818051 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:13.817647    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 22:39:14.817341 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:14.817303    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 22:39:14.817575 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:14.817360    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 22:39:15.817413 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:15.817374    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 22:39:15.817976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:15.817439    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 22:39:16.817746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:16.817711    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 22:39:16.818184 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:16.817773    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 22:39:17.822933 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:17.822892    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:17.823353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:17.823316    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:17.824147 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:17.824121    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:23.388434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:23.388398    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:39:23.400744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:23.399109    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:39:24.014000 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.013954    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:39:24.014303 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.014254    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" podUID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" containerName="maas-api" containerID="cri-o://824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e" gracePeriod=30
Nov 02 22:39:24.286237 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.286207    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:39:24.380602 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.380569    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\" (UniqueName: \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\") pod \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") "
Nov 02 22:39:24.380805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.380673    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\") pod \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") "
Nov 02 22:39:24.380805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.380707    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\") pod \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") "
Nov 02 22:39:24.380805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.380777    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\") pod \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\" (UID: \"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d\") "
Nov 02 22:39:24.381234 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.381205    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca" pod "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" (UID: "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:24.382924 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.382897    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs" pod "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" (UID: "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:39:24.383046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.382908    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls" pod "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" (UID: "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:39:24.383046 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.382959    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl" pod "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" (UID: "e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d"). InnerVolumeSpecName "kube-api-access-9qfnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:24.481669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.481630    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:24.481669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.481662    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:24.481669 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.481674    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:24.482140 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.481683    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfnl\" (UniqueName: \"kubernetes.io/projected/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d-kube-api-access-9qfnl\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:24.971697 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:24.971658    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:39:25.141445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.137751    4929 generic.go:356] "Generic (PLEG): container finished" podID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" containerID="824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e" exitCode=0
Nov 02 22:39:25.141445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.137895    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" event={"ID":"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d","Type":"ContainerDied","Data":"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"}
Nov 02 22:39:25.141445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.137950    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q" event={"ID":"e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d","Type":"ContainerDied","Data":"9096d9de01b1a38dabbd03afd60f31d9d09ecd8a95195e0de50a8ba9c056dace"}
Nov 02 22:39:25.141445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.137978    4929 scope.go:122] "RemoveContainer" containerID="824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"
Nov 02 22:39:25.141445 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.139101    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"
Nov 02 22:39:25.156832 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.156812    4929 scope.go:122] "RemoveContainer" containerID="824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"
Nov 02 22:39:25.157220 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:25.157187    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e\": container with ID starting with 824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e not found: ID does not exist" containerID="824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"
Nov 02 22:39:25.157367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.157235    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e"} err="failed to get container status \"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e\": rpc error: code = NotFound desc = could not find container \"824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e\": container with ID starting with 824796b18ac4b86ff3f9b0622102f566facdafb3f9fa0b41911a46dda8acf63e not found: ID does not exist"
Nov 02 22:39:25.159805 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.159781    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:39:25.164994 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:25.164972    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-555ba599-f8f49ff8d-ssv9q"]
Nov 02 22:39:26.802477 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:26.802443    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" path="/var/lib/kubelet/pods/e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d/volumes"
Nov 02 22:39:27.495024 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:27.494987    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:39:28.389169 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.389087    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" podUID="33c15709-2394-447c-bf66-2fbb99b5b5c2" containerName="payload-processing" containerID="cri-o://ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252" gracePeriod=25
Nov 02 22:39:28.399609 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.399574    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" podUID="1aafb070-c5b1-411c-ba1b-113832444f69" containerName="payload-pre-processing" containerID="cri-o://f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110" gracePeriod=25
Nov 02 22:39:28.663143 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.663117    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:39:28.683383 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.683344    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:39:28.724200 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724168    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\" (UniqueName: \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\") pod \"33c15709-2394-447c-bf66-2fbb99b5b5c2\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") "
Nov 02 22:39:28.724352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724220    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\" (UniqueName: \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\") pod \"1aafb070-c5b1-411c-ba1b-113832444f69\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") "
Nov 02 22:39:28.724352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724268    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\") pod \"33c15709-2394-447c-bf66-2fbb99b5b5c2\" (UID: \"33c15709-2394-447c-bf66-2fbb99b5b5c2\") "
Nov 02 22:39:28.724352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724318    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\") pod \"1aafb070-c5b1-411c-ba1b-113832444f69\" (UID: \"1aafb070-c5b1-411c-ba1b-113832444f69\") "
Nov 02 22:39:28.724764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724737    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume" pod "33c15709-2394-447c-bf66-2fbb99b5b5c2" (UID: "33c15709-2394-447c-bf66-2fbb99b5b5c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:28.724889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.724805    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume" pod "1aafb070-c5b1-411c-ba1b-113832444f69" (UID: "1aafb070-c5b1-411c-ba1b-113832444f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:28.726461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.726435    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf" pod "1aafb070-c5b1-411c-ba1b-113832444f69" (UID: "1aafb070-c5b1-411c-ba1b-113832444f69"). InnerVolumeSpecName "kube-api-access-mvrgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:28.726461 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.726443    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9" pod "33c15709-2394-447c-bf66-2fbb99b5b5c2" (UID: "33c15709-2394-447c-bf66-2fbb99b5b5c2"). InnerVolumeSpecName "kube-api-access-j4zb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:28.825902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.825872    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zb9\" (UniqueName: \"kubernetes.io/projected/33c15709-2394-447c-bf66-2fbb99b5b5c2-kube-api-access-j4zb9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:28.825902 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.825901    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvrgf\" (UniqueName: \"kubernetes.io/projected/1aafb070-c5b1-411c-ba1b-113832444f69-kube-api-access-mvrgf\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:28.826062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.825911    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c15709-2394-447c-bf66-2fbb99b5b5c2-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:28.826062 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:28.825920    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aafb070-c5b1-411c-ba1b-113832444f69-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:29.158746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.158709    4929 generic.go:356] "Generic (PLEG): container finished" podID="33c15709-2394-447c-bf66-2fbb99b5b5c2" containerID="ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252" exitCode=0
Nov 02 22:39:29.158960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.158771    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"
Nov 02 22:39:29.158960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.158806    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" event={"ID":"33c15709-2394-447c-bf66-2fbb99b5b5c2","Type":"ContainerDied","Data":"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"}
Nov 02 22:39:29.158960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.158886    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q" event={"ID":"33c15709-2394-447c-bf66-2fbb99b5b5c2","Type":"ContainerDied","Data":"72c4e4c3450331da457501b9c2e49588cdf4a44be68e15a96637a58c47dc4d6f"}
Nov 02 22:39:29.158960 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.158908    4929 scope.go:122] "RemoveContainer" containerID="ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"
Nov 02 22:39:29.160288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.160267    4929 generic.go:356] "Generic (PLEG): container finished" podID="1aafb070-c5b1-411c-ba1b-113832444f69" containerID="f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110" exitCode=0
Nov 02 22:39:29.160402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.160325    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"
Nov 02 22:39:29.160402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.160347    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" event={"ID":"1aafb070-c5b1-411c-ba1b-113832444f69","Type":"ContainerDied","Data":"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"}
Nov 02 22:39:29.160402 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.160383    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj" event={"ID":"1aafb070-c5b1-411c-ba1b-113832444f69","Type":"ContainerDied","Data":"e2570686d6927801805e0f21a8d1cd2c88e5654733c446ecae70d084daeba9a7"}
Nov 02 22:39:29.171641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.171619    4929 scope.go:122] "RemoveContainer" containerID="ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"
Nov 02 22:39:29.171937 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:29.171915    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252\": container with ID starting with ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252 not found: ID does not exist" containerID="ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"
Nov 02 22:39:29.172035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.171947    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252"} err="failed to get container status \"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252\": rpc error: code = NotFound desc = could not find container \"ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252\": container with ID starting with ac109b3656013d71797562b1b3c716a80bb7e7672a266e08b1628b4ab5ccc252 not found: ID does not exist"
Nov 02 22:39:29.172035 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.171964    4929 scope.go:122] "RemoveContainer" containerID="f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"
Nov 02 22:39:29.178493 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.178464    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:39:29.180100 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.180073    4929 scope.go:122] "RemoveContainer" containerID="f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"
Nov 02 22:39:29.180452 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:29.180421    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110\": container with ID starting with f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110 not found: ID does not exist" containerID="f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"
Nov 02 22:39:29.180594 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.180461    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110"} err="failed to get container status \"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110\": rpc error: code = NotFound desc = could not find container \"f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110\": container with ID starting with f5c6e9f0558b07f04d45813f764da777d5562b6f33b1019d51b1134d8b2af110 not found: ID does not exist"
Nov 02 22:39:29.181950 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.181931    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-555ba599-69995448cc-4cr5q"]
Nov 02 22:39:29.191527 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.191505    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:39:29.193357 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.193330    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-555ba599-7bddb8546c-248dj"]
Nov 02 22:39:29.594638 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.594550    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:39:29.595131 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:29.594756    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" containerID="cri-o://73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e" gracePeriod=30
Nov 02 22:39:30.803008 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:30.802976    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aafb070-c5b1-411c-ba1b-113832444f69" path="/var/lib/kubelet/pods/1aafb070-c5b1-411c-ba1b-113832444f69/volumes"
Nov 02 22:39:30.803400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:30.803344    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c15709-2394-447c-bf66-2fbb99b5b5c2" path="/var/lib/kubelet/pods/33c15709-2394-447c-bf66-2fbb99b5b5c2/volumes"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.313632    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.314995    4929 cpu_manager.go:414] "containerMap: removing container" podUID="1aafb070-c5b1-411c-ba1b-113832444f69" containerName="payload-pre-processing"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315019    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aafb070-c5b1-411c-ba1b-113832444f69" containerName="payload-pre-processing"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315060    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315068    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315098    4929 cpu_manager.go:414] "containerMap: removing container" podUID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315107    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315123    4929 cpu_manager.go:414] "containerMap: removing container" podUID="33c15709-2394-447c-bf66-2fbb99b5b5c2" containerName="payload-processing"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315131    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c15709-2394-447c-bf66-2fbb99b5b5c2" containerName="payload-processing"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315257    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e8b8b5-8596-43d3-90b2-8db8f8e72c9d" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315275    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac17491a-49fb-42c8-873d-c4d4a457a2fd" containerName="maas-api"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315285    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c15709-2394-447c-bf66-2fbb99b5b5c2" containerName="payload-processing"
Nov 02 22:39:31.316631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.315298    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aafb070-c5b1-411c-ba1b-113832444f69" containerName="payload-pre-processing"
Nov 02 22:39:31.320703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.320274    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:39:31.320703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.320398    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.329907 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.329883    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-30c892-openshift-default-dockercfg-p9hqp\""
Nov 02 22:39:31.452246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452209    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452246 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452246    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452276    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452565 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452359    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452570    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452609    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452710 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452673    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452725    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.452879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.452802    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6qj\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553653 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553613    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553675    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553713    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6qj\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553754    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553778    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553818    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.553864 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553851    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.553943    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554146    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554189    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554199    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554429 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554269    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554502 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554475    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.554678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.554656    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.556664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.556639    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.556882 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.556864    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.562758 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.562732    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.562866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.562834    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6qj\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\") pod \"e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") " pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.640321 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.640285    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:31.925664 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.925592    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:39:31.928294 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:39:31.928266    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd36643_4915_4d41_b0e4_a79c1bdbd6bd.slice/crio-177c4d5f0324f940f8f6f8cedb38eb13b824118907fce8be6b4d1ad1eaefb7ca WatchSource:0}: Error finding container 177c4d5f0324f940f8f6f8cedb38eb13b824118907fce8be6b4d1ad1eaefb7ca: Status 404 returned error can't find the container with id 177c4d5f0324f940f8f6f8cedb38eb13b824118907fce8be6b4d1ad1eaefb7ca
Nov 02 22:39:31.930956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.930924    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:31.931027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.930991    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:31.931027 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:31.931021    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:32.178667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.178571    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerStarted","Data":"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"}
Nov 02 22:39:32.178667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.178634    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerStarted","Data":"177c4d5f0324f940f8f6f8cedb38eb13b824118907fce8be6b4d1ad1eaefb7ca"}
Nov 02 22:39:32.199350 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.199292    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podStartSLOduration=1.19927555 podStartE2EDuration="1.19927555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:32.197732004 +0000 UTC m=+2509.505822814" watchObservedRunningTime="2026-11-02 22:39:32.19927555 +0000 UTC m=+2509.507366354"
Nov 02 22:39:32.640976 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.640937    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:32.646303 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.646270    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:32.826471 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.826433    4929 patch_prober.go:29] interesting pod/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:39:32.826663 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:32.826488    4929 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:39:33.182818 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:33.182785    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:33.183641 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:33.183620    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:34.852854 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.852827    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:34.987142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987042    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987142 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987116    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987151    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987352 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987193    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987452 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987420    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987584 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987558    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987622    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:34.987678 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987640    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987683    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987793 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987754    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\") pod \"30454237-881c-4c13-979e-1bbfb8deb44a\" (UID: \"30454237-881c-4c13-979e-1bbfb8deb44a\") "
Nov 02 22:39:34.987926 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.987802    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:39:34.988083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988018    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:39:34.988083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988034    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:39:34.988272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988248    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:39:34.988272 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988261    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:34.988404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988282    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:34.988404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988298    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:34.988404 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.988312    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30454237-881c-4c13-979e-1bbfb8deb44a-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:34.989567 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.989518    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:34.989675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.989594    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "kube-api-access-kjh2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:34.989675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.989622    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:39:34.989745 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:34.989662    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo" pod "30454237-881c-4c13-979e-1bbfb8deb44a" (UID: "30454237-881c-4c13-979e-1bbfb8deb44a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:39:35.089495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.089454    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjh2s\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-kube-api-access-kjh2s\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:35.089495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.089490    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30454237-881c-4c13-979e-1bbfb8deb44a-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:35.089495 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.089499    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:35.089791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.089509    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30454237-881c-4c13-979e-1bbfb8deb44a-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:35.089791 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.089518    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30454237-881c-4c13-979e-1bbfb8deb44a-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:35.194456 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.194416    4929 generic.go:356] "Generic (PLEG): container finished" podID="30454237-881c-4c13-979e-1bbfb8deb44a" containerID="73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e" exitCode=0
Nov 02 22:39:35.194648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.194475    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"
Nov 02 22:39:35.194648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.194497    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerDied","Data":"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"}
Nov 02 22:39:35.194648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.194571    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4" event={"ID":"30454237-881c-4c13-979e-1bbfb8deb44a","Type":"ContainerDied","Data":"5632fd34d480aff785990bf9dbf54491fcb0ca310e138c303cb9353f018d8df6"}
Nov 02 22:39:35.194648 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.194595    4929 scope.go:122] "RemoveContainer" containerID="73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"
Nov 02 22:39:35.204251 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.204228    4929 scope.go:122] "RemoveContainer" containerID="decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"
Nov 02 22:39:35.212230 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.212211    4929 scope.go:122] "RemoveContainer" containerID="73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"
Nov 02 22:39:35.212515 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:35.212491    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e\": container with ID starting with 73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e not found: ID does not exist" containerID="73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"
Nov 02 22:39:35.212631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.212521    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e"} err="failed to get container status \"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e\": rpc error: code = NotFound desc = could not find container \"73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e\": container with ID starting with 73aa2eef11ce0dbd8abb7447af23977d77ef6dcdfd3b755300d44129f950885e not found: ID does not exist"
Nov 02 22:39:35.212631 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.212564    4929 scope.go:122] "RemoveContainer" containerID="decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"
Nov 02 22:39:35.212863 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:35.212847    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d\": container with ID starting with decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d not found: ID does not exist" containerID="decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"
Nov 02 22:39:35.212922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.212866    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d"} err="failed to get container status \"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d\": rpc error: code = NotFound desc = could not find container \"decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d\": container with ID starting with decfab8863dcbb77228258c39516db2c58ca42599c88a5bcf64d5230213d897d not found: ID does not exist"
Nov 02 22:39:35.216831 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.216809    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:39:35.220294 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.220270    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-555ba599-openshift-default-bf6d7676c-h5qn4"]
Nov 02 22:39:35.516029 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.515990    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:39:35.516487 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516468    4929 cpu_manager.go:414] "containerMap: removing container" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.516616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516489    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.516616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516505    4929 cpu_manager.go:414] "containerMap: removing container" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.516616 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516513    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.516861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516635    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.516861 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.516649    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" containerName="istio-proxy"
Nov 02 22:39:35.521439 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.521409    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.530957 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.530902    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:39:35.534823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.534794    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-30c892-serving-cert\""
Nov 02 22:39:35.593295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.593259    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.593450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.593392    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrqr\" (UniqueName: \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.593450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.593433    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.593564 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.593488    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.625702 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.625661    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:39:35.629240 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.629216    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.637283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.637260    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-30c892-dockercfg-br9vv\""
Nov 02 22:39:35.641255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.641228    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:39:35.641377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.641362    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-30c892\""
Nov 02 22:39:35.670881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.670847    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:39:35.674434 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.674410    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.684572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.683821    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694125    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shl4\" (UniqueName: \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694204    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrqr\" (UniqueName: \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694260    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694334    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694365    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.694904 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.694395    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.696162 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.696132    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.697353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.697327    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.697881 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.697857    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.703087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.703066    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrqr\" (UniqueName: \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\") pod \"maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.796017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.795916    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.796017 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.795982    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.796232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.796113    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shl4\" (UniqueName: \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.796232 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.796181    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hms9\" (UniqueName: \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.797033 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.797008    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.804927 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.804892    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shl4\" (UniqueName: \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\") pod \"payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.839788 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.839756    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:35.897158 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.897126    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.897637 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.897250    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hms9\" (UniqueName: \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.901522 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.898048    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.908490 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.908465    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hms9\" (UniqueName: \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\") pod \"payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:35.944172 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.944140    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:35.990504 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:35.990474    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:36.116247 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:36.116219    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:39:36.118412 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:39:36.118383    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode458ec8f_0551_4089_b1ab_ce46fab610f7.slice/crio-27b76663647b0395020cf6037e7acf52cba8253ad720fe5117a6eca303fceda7 WatchSource:0}: Error finding container 27b76663647b0395020cf6037e7acf52cba8253ad720fe5117a6eca303fceda7: Status 404 returned error can't find the container with id 27b76663647b0395020cf6037e7acf52cba8253ad720fe5117a6eca303fceda7
Nov 02 22:39:36.205792 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:36.204778    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" event={"ID":"e458ec8f-0551-4089-b1ab-ce46fab610f7","Type":"ContainerStarted","Data":"27b76663647b0395020cf6037e7acf52cba8253ad720fe5117a6eca303fceda7"}
Nov 02 22:39:36.223798 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:36.223773    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:39:36.225857 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:39:36.225831    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047ee1f9_e492_4dc1_ac09_07586219acc0.slice/crio-350d659d450ea3ef916dcbdcbba04036f1369c444cd790259f78227f1bc494e2 WatchSource:0}: Error finding container 350d659d450ea3ef916dcbdcbba04036f1369c444cd790259f78227f1bc494e2: Status 404 returned error can't find the container with id 350d659d450ea3ef916dcbdcbba04036f1369c444cd790259f78227f1bc494e2
Nov 02 22:39:36.298574 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:36.298548    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:39:36.301127 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:39:36.301102    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64be08c4_ce11_49c6_9d1e_0abccb1a76ad.slice/crio-0f147b419ab0087186995aef0e60d23b9d60e7ba30a1ef39edcc85b6554a64ff WatchSource:0}: Error finding container 0f147b419ab0087186995aef0e60d23b9d60e7ba30a1ef39edcc85b6554a64ff: Status 404 returned error can't find the container with id 0f147b419ab0087186995aef0e60d23b9d60e7ba30a1ef39edcc85b6554a64ff
Nov 02 22:39:36.805286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:36.805250    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30454237-881c-4c13-979e-1bbfb8deb44a" path="/var/lib/kubelet/pods/30454237-881c-4c13-979e-1bbfb8deb44a/volumes"
Nov 02 22:39:37.215808 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.215772    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" event={"ID":"64be08c4-ce11-49c6-9d1e-0abccb1a76ad","Type":"ContainerStarted","Data":"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"}
Nov 02 22:39:37.216307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.215833    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" event={"ID":"64be08c4-ce11-49c6-9d1e-0abccb1a76ad","Type":"ContainerStarted","Data":"0f147b419ab0087186995aef0e60d23b9d60e7ba30a1ef39edcc85b6554a64ff"}
Nov 02 22:39:37.216307 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.215953    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:37.217366 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.217343    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" event={"ID":"e458ec8f-0551-4089-b1ab-ce46fab610f7","Type":"ContainerStarted","Data":"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"}
Nov 02 22:39:37.217556 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.217385    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:37.218865 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.218845    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" event={"ID":"047ee1f9-e492-4dc1-ac09-07586219acc0","Type":"ContainerStarted","Data":"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"}
Nov 02 22:39:37.218978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.218886    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" event={"ID":"047ee1f9-e492-4dc1-ac09-07586219acc0","Type":"ContainerStarted","Data":"350d659d450ea3ef916dcbdcbba04036f1369c444cd790259f78227f1bc494e2"}
Nov 02 22:39:37.218978 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.218924    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:37.231251 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.231204    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" podStartSLOduration=2.231193414 podStartE2EDuration="2.231193414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:37.23061858 +0000 UTC m=+2514.538709385" watchObservedRunningTime="2026-11-02 22:39:37.231193414 +0000 UTC m=+2514.539284215"
Nov 02 22:39:37.246898 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.246852    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" podStartSLOduration=1.7760743479999999 podStartE2EDuration="2.246838525s" totalImagesPullingTime="470.764177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:37.246077929 +0000 UTC m=+2514.554168750" watchObservedRunningTime="2026-11-02 22:39:37.246838525 +0000 UTC m=+2514.554929327"
Nov 02 22:39:37.263582 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.263517    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" podStartSLOduration=2.263504102 podStartE2EDuration="2.263504102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:37.262169682 +0000 UTC m=+2514.570260477" watchObservedRunningTime="2026-11-02 22:39:37.263504102 +0000 UTC m=+2514.571594930"
Nov 02 22:39:37.945962 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:37.945929    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:39:43.231040 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:43.231008    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:39:47.295121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.295018    4929 generic.go:356] "Generic (PLEG): container finished" podID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerID="b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1" exitCode=0
Nov 02 22:39:47.295121 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.295095    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerDied","Data":"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"}
Nov 02 22:39:47.295634 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.295508    4929 scope.go:122] "RemoveContainer" containerID="b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"
Nov 02 22:39:47.296740 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.296711    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:47.296819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.296779    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:47.296819 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:47.296810    4929 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 22:39:48.183048 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.183010    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:48.228003 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.227970    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:39:48.228207 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.228052    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:39:48.302839 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.302797    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerStarted","Data":"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"}
Nov 02 22:39:48.644947 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.644912    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:48.665729 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.665689    4929 patch_prober.go:29] interesting pod/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 22:39:48.666290 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:48.666259    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 22:39:49.659559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:49.656962    4929 patch_prober.go:29] interesting pod/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 22:39:49.659559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:49.657027    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 22:39:50.656558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:50.646951    4929 patch_prober.go:29] interesting pod/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 22:39:50.656558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:50.647014    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 22:39:51.652559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:51.642942    4929 patch_prober.go:29] interesting pod/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 22:39:51.652559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:51.643006    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 22:39:52.643068 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:52.643025    4929 patch_prober.go:29] interesting pod/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 22:39:52.643384 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:52.643354    4929 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 22:39:53.644955 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:53.644919    4929 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:53.645400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:53.645310    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:53.646023 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:39:53.646002    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:39:57.495696 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:39:57.495658    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:40:27.496785 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:40:27.496702    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:40:57.496555 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:40:57.496501    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:41:19.327975 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:19.327930    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:41:19.335256 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:19.335219    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:41:24.328727 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.328673    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" podUID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" containerName="payload-processing" containerID="cri-o://9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864" gracePeriod=25
Nov 02 22:41:24.336283 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.336227    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" podUID="047ee1f9-e492-4dc1-ac09-07586219acc0" containerName="payload-pre-processing" containerID="cri-o://3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8" gracePeriod=25
Nov 02 22:41:24.599578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.599549    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:41:24.632824 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.632784    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:41:24.643979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.643942    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\") pod \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") "
Nov 02 22:41:24.644153 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.644103    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\" (UniqueName: \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\") pod \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\" (UID: \"64be08c4-ce11-49c6-9d1e-0abccb1a76ad\") "
Nov 02 22:41:24.644339 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.644317    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume" pod "64be08c4-ce11-49c6-9d1e-0abccb1a76ad" (UID: "64be08c4-ce11-49c6-9d1e-0abccb1a76ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:41:24.644440 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.644428    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:24.646221 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.646193    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9" pod "64be08c4-ce11-49c6-9d1e-0abccb1a76ad" (UID: "64be08c4-ce11-49c6-9d1e-0abccb1a76ad"). InnerVolumeSpecName "kube-api-access-6hms9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:41:24.744823 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.744781    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\" (UniqueName: \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\") pod \"047ee1f9-e492-4dc1-ac09-07586219acc0\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") "
Nov 02 22:41:24.745010 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.744846    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\") pod \"047ee1f9-e492-4dc1-ac09-07586219acc0\" (UID: \"047ee1f9-e492-4dc1-ac09-07586219acc0\") "
Nov 02 22:41:24.745116 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.745103    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hms9\" (UniqueName: \"kubernetes.io/projected/64be08c4-ce11-49c6-9d1e-0abccb1a76ad-kube-api-access-6hms9\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:24.745211 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.745188    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume" pod "047ee1f9-e492-4dc1-ac09-07586219acc0" (UID: "047ee1f9-e492-4dc1-ac09-07586219acc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:41:24.746995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.746972    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4" pod "047ee1f9-e492-4dc1-ac09-07586219acc0" (UID: "047ee1f9-e492-4dc1-ac09-07586219acc0"). InnerVolumeSpecName "kube-api-access-6shl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:41:24.759188 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.759155    4929 generic.go:356] "Generic (PLEG): container finished" podID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" containerID="9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864" exitCode=0
Nov 02 22:41:24.759344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.759197    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"
Nov 02 22:41:24.759344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.759238    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" event={"ID":"64be08c4-ce11-49c6-9d1e-0abccb1a76ad","Type":"ContainerDied","Data":"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"}
Nov 02 22:41:24.759344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.759296    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5" event={"ID":"64be08c4-ce11-49c6-9d1e-0abccb1a76ad","Type":"ContainerDied","Data":"0f147b419ab0087186995aef0e60d23b9d60e7ba30a1ef39edcc85b6554a64ff"}
Nov 02 22:41:24.759344 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.759329    4929 scope.go:122] "RemoveContainer" containerID="9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"
Nov 02 22:41:24.760613 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.760591    4929 generic.go:356] "Generic (PLEG): container finished" podID="047ee1f9-e492-4dc1-ac09-07586219acc0" containerID="3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8" exitCode=0
Nov 02 22:41:24.760714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.760621    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" event={"ID":"047ee1f9-e492-4dc1-ac09-07586219acc0","Type":"ContainerDied","Data":"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"}
Nov 02 22:41:24.760714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.760647    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"
Nov 02 22:41:24.760714 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.760674    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx" event={"ID":"047ee1f9-e492-4dc1-ac09-07586219acc0","Type":"ContainerDied","Data":"350d659d450ea3ef916dcbdcbba04036f1369c444cd790259f78227f1bc494e2"}
Nov 02 22:41:24.768498 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.768471    4929 scope.go:122] "RemoveContainer" containerID="9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"
Nov 02 22:41:24.768833 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:24.768805    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864\": container with ID starting with 9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864 not found: ID does not exist" containerID="9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"
Nov 02 22:41:24.768922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.768846    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864"} err="failed to get container status \"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864\": rpc error: code = NotFound desc = could not find container \"9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864\": container with ID starting with 9dc7c6beaddbf1fc712864be763c06a7c76a559df7ebad6c0a73d7c473d25864 not found: ID does not exist"
Nov 02 22:41:24.768922 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.768868    4929 scope.go:122] "RemoveContainer" containerID="3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"
Nov 02 22:41:24.777968 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.777944    4929 scope.go:122] "RemoveContainer" containerID="3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"
Nov 02 22:41:24.778296 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:24.778271    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8\": container with ID starting with 3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8 not found: ID does not exist" containerID="3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"
Nov 02 22:41:24.778377 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.778308    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8"} err="failed to get container status \"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8\": rpc error: code = NotFound desc = could not find container \"3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8\": container with ID starting with 3330edafd87fd9c4d739b469644e13af59b4493cad8c495a739fc9e98d6539b8 not found: ID does not exist"
Nov 02 22:41:24.785814 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.785788    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:41:24.788036 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.787968    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-30c892-79c9c884bc-gcqjx"]
Nov 02 22:41:24.804390 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.804354    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047ee1f9-e492-4dc1-ac09-07586219acc0" path="/var/lib/kubelet/pods/047ee1f9-e492-4dc1-ac09-07586219acc0/volumes"
Nov 02 22:41:24.804783 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.804762    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:41:24.804885 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.804802    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-30c892-5c9978b45-vsmz5"]
Nov 02 22:41:24.846043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.845950    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shl4\" (UniqueName: \"kubernetes.io/projected/047ee1f9-e492-4dc1-ac09-07586219acc0-kube-api-access-6shl4\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:24.846043 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:24.845987    4929 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047ee1f9-e492-4dc1-ac09-07586219acc0-plugins-config-volume\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:25.215991 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.215956    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:41:25.216225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.216183    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" podUID="e458ec8f-0551-4089-b1ab-ce46fab610f7" containerName="maas-api" containerID="cri-o://219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4" gracePeriod=30
Nov 02 22:41:25.474073 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.473968    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:41:25.554444 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.554386    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\") pod \"e458ec8f-0551-4089-b1ab-ce46fab610f7\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") "
Nov 02 22:41:25.554675 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.554605    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\" (UniqueName: \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\") pod \"e458ec8f-0551-4089-b1ab-ce46fab610f7\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") "
Nov 02 22:41:25.554759 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.554732    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\") pod \"e458ec8f-0551-4089-b1ab-ce46fab610f7\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") "
Nov 02 22:41:25.554822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.554771    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\") pod \"e458ec8f-0551-4089-b1ab-ce46fab610f7\" (UID: \"e458ec8f-0551-4089-b1ab-ce46fab610f7\") "
Nov 02 22:41:25.555225 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.555191    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca" pod "e458ec8f-0551-4089-b1ab-ce46fab610f7" (UID: "e458ec8f-0551-4089-b1ab-ce46fab610f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:41:25.557080 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.557051    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls" pod "e458ec8f-0551-4089-b1ab-ce46fab610f7" (UID: "e458ec8f-0551-4089-b1ab-ce46fab610f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:41:25.557187 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.557156    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr" pod "e458ec8f-0551-4089-b1ab-ce46fab610f7" (UID: "e458ec8f-0551-4089-b1ab-ce46fab610f7"). InnerVolumeSpecName "kube-api-access-zxrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:41:25.557274 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.557258    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs" pod "e458ec8f-0551-4089-b1ab-ce46fab610f7" (UID: "e458ec8f-0551-4089-b1ab-ce46fab610f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:41:25.655952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.655910    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrqr\" (UniqueName: \"kubernetes.io/projected/e458ec8f-0551-4089-b1ab-ce46fab610f7-kube-api-access-zxrqr\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:25.655952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.655946    4929 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-maas-api-tls\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:25.655952 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.655958    4929 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e458ec8f-0551-4089-b1ab-ce46fab610f7-openshift-service-ca\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:25.656212 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.655967    4929 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e458ec8f-0551-4089-b1ab-ce46fab610f7-metrics-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:25.767614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.767495    4929 generic.go:356] "Generic (PLEG): container finished" podID="e458ec8f-0551-4089-b1ab-ce46fab610f7" containerID="219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4" exitCode=0
Nov 02 22:41:25.767614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.767565    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"
Nov 02 22:41:25.767614 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.767579    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" event={"ID":"e458ec8f-0551-4089-b1ab-ce46fab610f7","Type":"ContainerDied","Data":"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"}
Nov 02 22:41:25.767905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.767627    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx" event={"ID":"e458ec8f-0551-4089-b1ab-ce46fab610f7","Type":"ContainerDied","Data":"27b76663647b0395020cf6037e7acf52cba8253ad720fe5117a6eca303fceda7"}
Nov 02 22:41:25.767905 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.767654    4929 scope.go:122] "RemoveContainer" containerID="219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"
Nov 02 22:41:25.777474 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.777456    4929 scope.go:122] "RemoveContainer" containerID="219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"
Nov 02 22:41:25.777777 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:25.777753    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4\": container with ID starting with 219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4 not found: ID does not exist" containerID="219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"
Nov 02 22:41:25.777884 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.777785    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4"} err="failed to get container status \"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4\": rpc error: code = NotFound desc = could not find container \"219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4\": container with ID starting with 219371fff526bf30dc7550686ba55acba24a6ffc46675ff09df304d097948cd4 not found: ID does not exist"
Nov 02 22:41:25.790558 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.790513    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:41:25.796132 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:25.796105    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-30c892-f7c69f58c-r24mx"]
Nov 02 22:41:26.802554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:26.802512    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" path="/var/lib/kubelet/pods/64be08c4-ce11-49c6-9d1e-0abccb1a76ad/volumes"
Nov 02 22:41:26.802979 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:26.802965    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e458ec8f-0551-4089-b1ab-ce46fab610f7" path="/var/lib/kubelet/pods/e458ec8f-0551-4089-b1ab-ce46fab610f7/volumes"
Nov 02 22:41:27.008386 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:27.008348    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:41:27.495399 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:27.495364    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:41:31.763654 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:31.763603    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:41:31.764175 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:31.763911    4929 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy" containerID="cri-o://f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1" gracePeriod=30
Nov 02 22:41:33.781764 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:33.781724    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:41:37.078956 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.078928    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:41:37.188472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188433    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188472 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188478    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188508    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188559    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188726 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188628    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188761    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188824    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188859    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.188879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188851    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:41:37.189103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188863    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:41:37.189103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.188888    4929 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\") pod \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\" (UID: \"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd\") "
Nov 02 22:41:37.189103 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189087    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:41:37.189268 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189079    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:41:37.189353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189331    4929 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-certs\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.189353 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189354    4929 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-data\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.189554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189367    4929 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-credential-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.189554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189380    4929 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-workload-socket\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.189554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.189401    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:41:37.191333 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.191306    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:41:37.191457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.191331    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:41:37.191457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.191363    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "kube-api-access-4l6qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:41:37.191457 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.191363    4929 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo" pod "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" (UID: "2fd36643-4915-4d41-b0e4-a79c1bdbd6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:41:37.290521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.290481    4929 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-token\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.290521 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.290514    4929 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istiod-ca-cert\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.290752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.290526    4929 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l6qj\" (UniqueName: \"kubernetes.io/projected/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-kube-api-access-4l6qj\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.290752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.290564    4929 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-envoy\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.290752 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.290573    4929 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd-istio-podinfo\") on node \"ip-10-0-131-57.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 22:41:37.856171 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.856137    4929 generic.go:356] "Generic (PLEG): container finished" podID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerID="f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1" exitCode=0
Nov 02 22:41:37.856336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.856190    4929 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"
Nov 02 22:41:37.856336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.856225    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerDied","Data":"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"}
Nov 02 22:41:37.856336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.856284    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x" event={"ID":"2fd36643-4915-4d41-b0e4-a79c1bdbd6bd","Type":"ContainerDied","Data":"177c4d5f0324f940f8f6f8cedb38eb13b824118907fce8be6b4d1ad1eaefb7ca"}
Nov 02 22:41:37.856336 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.856306    4929 scope.go:122] "RemoveContainer" containerID="f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"
Nov 02 22:41:37.871747 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.871720    4929 scope.go:122] "RemoveContainer" containerID="b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"
Nov 02 22:41:37.881740 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.881701    4929 scope.go:122] "RemoveContainer" containerID="f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"
Nov 02 22:41:37.882252 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:37.882188    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1\": container with ID starting with f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1 not found: ID does not exist" containerID="f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"
Nov 02 22:41:37.882342 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.882309    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1"} err="failed to get container status \"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1\": rpc error: code = NotFound desc = could not find container \"f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1\": container with ID starting with f5be9b1105c5a52faa90b7c885e4b50b4d2e1dacbe79c84eba67191449a7aed1 not found: ID does not exist"
Nov 02 22:41:37.882415 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.882353    4929 scope.go:122] "RemoveContainer" containerID="b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"
Nov 02 22:41:37.882894 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:37.882830    4929 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1\": container with ID starting with b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1 not found: ID does not exist" containerID="b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"
Nov 02 22:41:37.882894 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.882872    4929 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1"} err="failed to get container status \"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1\": rpc error: code = NotFound desc = could not find container \"b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1\": container with ID starting with b8db089fcd080c5d71560f2ae0eee8e408be0f7143320da49664c9c092f2b3f1 not found: ID does not exist"
Nov 02 22:41:37.885446 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.885425    4929 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:41:37.890650 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:37.890627    4929 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-30c892-openshift-default-785d584d49-pfw9x"]
Nov 02 22:41:38.803071 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:38.803026    4929 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" path="/var/lib/kubelet/pods/2fd36643-4915-4d41-b0e4-a79c1bdbd6bd/volumes"
Nov 02 22:41:38.877289 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:38.877253    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:41:53.938572 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:41:53.938525    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:41:57.617555 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:41:57.613741    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:42:20.578114 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:20.578071    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpmtr"]
Nov 02 22:42:27.496027 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:42:27.495979    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:42:32.023741 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:32.023705    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:42:33.520605 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:33.520565    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b994c7666-hr4rf_7672b2f9-0101-4ad1-90ae-c73989202d13/maas-api/0.log"
Nov 02 22:42:37.211483 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:37.211453    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpmtr_e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3/limitador/0.log"
Nov 02 22:42:38.711667 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:38.711637    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-mtdsj_2cf18102-5f02-4e2c-b46b-043328bb8f66/kube-auth-proxy/0.log"
Nov 02 22:42:38.884020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:38.883987    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-2zhcx_5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90/istio-proxy/0.log"
Nov 02 22:42:39.061213 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:39.061132    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-gjn6k_4a73ff87-2b7d-4c69-82e2-495f4b866345/istio-proxy/0.log"
Nov 02 22:42:39.243781 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:39.243750    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-j6cnb_047c4566-edc7-453d-bd8c-d14867259b93/istio-proxy/0.log"
Nov 02 22:42:39.920775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:39.920737    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-wjhsd_e8cee094-9f9e-4b00-b658-431c6abdccc2/istio-proxy/0.log"
Nov 02 22:42:40.988300 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:40.988263    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:42:41.157603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:41.157566    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8469b8f4f5-bf9r8_327d452e-cee7-493d-bd9a-81dc5b5b42e2/router/0.log"
Nov 02 22:42:42.067295 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:42.067262    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj_bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70/main/0.log"
Nov 02 22:42:42.612097 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:42.612061    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-svvlm_7a4a8e4d-2131-4854-9d0b-5010dae227bb/main/0.log"
Nov 02 22:42:42.782083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:42.782048    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc_8b76db34-a58f-4b26-91c1-d5c678aec811/main/0.log"
Nov 02 22:42:43.293623 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:43.293574    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w_c137d510-9d3e-4e3b-94ec-8daf581918f9/main/0.log"
Nov 02 22:42:45.692566 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:45.692520    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:42:45.697603 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:45.697564    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:42:53.939712 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:53.939678    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:42:55.257396 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:55.257359    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b994c7666-hr4rf_7672b2f9-0101-4ad1-90ae-c73989202d13/maas-api/0.log"
Nov 02 22:42:57.495810 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:42:57.495773    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:42:58.550468 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:58.550433    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpmtr_e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3/limitador/0.log"
Nov 02 22:42:59.802778 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:59.802744    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-mtdsj_2cf18102-5f02-4e2c-b46b-043328bb8f66/kube-auth-proxy/0.log"
Nov 02 22:42:59.988455 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:42:59.988426    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-2zhcx_5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90/istio-proxy/0.log"
Nov 02 22:43:00.170394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:00.170356    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-gjn6k_4a73ff87-2b7d-4c69-82e2-495f4b866345/istio-proxy/0.log"
Nov 02 22:43:00.339050 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:00.339020    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-j6cnb_047c4566-edc7-453d-bd8c-d14867259b93/istio-proxy/0.log"
Nov 02 22:43:01.077934 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:01.077900    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-wjhsd_e8cee094-9f9e-4b00-b658-431c6abdccc2/istio-proxy/0.log"
Nov 02 22:43:02.136676 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:02.136636    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:43:02.317178 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:02.317140    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8469b8f4f5-bf9r8_327d452e-cee7-493d-bd9a-81dc5b5b42e2/router/0.log"
Nov 02 22:43:02.803392 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:02.803364    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qhhrj_bd8cdbfe-4ab5-4aa8-84de-f25c7a6bbb70/main/0.log"
Nov 02 22:43:03.344328 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:03.344297    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-svvlm_7a4a8e4d-2131-4854-9d0b-5010dae227bb/main/0.log"
Nov 02 22:43:03.521921 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:03.521887    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qkbpc_8b76db34-a58f-4b26-91c1-d5c678aec811/main/0.log"
Nov 02 22:43:04.064744 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:04.064697    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l9k8w_c137d510-9d3e-4e3b-94ec-8daf581918f9/main/0.log"
Nov 02 22:43:27.494075 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:43:27.493958    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:43:40.101020 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:40.100982    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:43:40.343643 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:40.343607    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:43:41.052679 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:41.052636    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:43:41.224851 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:43:41.224821    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:43:57.495832 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:43:57.495792    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:44:25.408362 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:25.408324    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:44:25.682261 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:25.682172    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68bfb5556c-nz4xl_7a15bfe7-9471-4d23-9fff-45014a560935/manager/0.log"
Nov 02 22:44:26.388316 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.388284    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:44:26.571767 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.571735    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:44:26.861255 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861220    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zc4v/must-gather-r298n"]
Nov 02 22:44:26.861625 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861610    4929 cpu_manager.go:414] "containerMap: removing container" podUID="047ee1f9-e492-4dc1-ac09-07586219acc0" containerName="payload-pre-processing"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861625    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047ee1f9-e492-4dc1-ac09-07586219acc0" containerName="payload-pre-processing"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861639    4929 cpu_manager.go:414] "containerMap: removing container" podUID="e458ec8f-0551-4089-b1ab-ce46fab610f7" containerName="maas-api"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861645    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e458ec8f-0551-4089-b1ab-ce46fab610f7" containerName="maas-api"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861667    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861673    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861680    4929 cpu_manager.go:414] "containerMap: removing container" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861685    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861694    4929 cpu_manager.go:414] "containerMap: removing container" podUID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" containerName="payload-processing"
Nov 02 22:44:26.861700 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861699    4929 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" containerName="payload-processing"
Nov 02 22:44:26.861995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861760    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="64be08c4-ce11-49c6-9d1e-0abccb1a76ad" containerName="payload-processing"
Nov 02 22:44:26.861995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861772    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="047ee1f9-e492-4dc1-ac09-07586219acc0" containerName="payload-pre-processing"
Nov 02 22:44:26.861995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861778    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="e458ec8f-0551-4089-b1ab-ce46fab610f7" containerName="maas-api"
Nov 02 22:44:26.861995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861786    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.861995 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.861792    4929 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd36643-4915-4d41-b0e4-a79c1bdbd6bd" containerName="istio-proxy"
Nov 02 22:44:26.865044 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.865025    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:26.869879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.869855    4929 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7zc4v\"/\"default-dockercfg-pw2c6\""
Nov 02 22:44:26.869879 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.869855    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zc4v\"/\"openshift-service-ca.crt\""
Nov 02 22:44:26.870041 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.869865    4929 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zc4v\"/\"kube-root-ca.crt\""
Nov 02 22:44:26.879022 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.878995    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/must-gather-r298n"]
Nov 02 22:44:26.937076 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.937034    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9dg\" (UniqueName: \"kubernetes.io/projected/1fe3f80b-76b0-4690-b3a0-17f6917c4615-kube-api-access-lw9dg\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:26.937260 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:26.937163    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1fe3f80b-76b0-4690-b3a0-17f6917c4615-must-gather-output\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.038578 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.038528    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9dg\" (UniqueName: \"kubernetes.io/projected/1fe3f80b-76b0-4690-b3a0-17f6917c4615-kube-api-access-lw9dg\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.038775 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.038612    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1fe3f80b-76b0-4690-b3a0-17f6917c4615-must-gather-output\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.038942 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.038925    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1fe3f80b-76b0-4690-b3a0-17f6917c4615-must-gather-output\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.046826 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.046798    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9dg\" (UniqueName: \"kubernetes.io/projected/1fe3f80b-76b0-4690-b3a0-17f6917c4615-kube-api-access-lw9dg\") pod \"must-gather-r298n\" (UID: \"1fe3f80b-76b0-4690-b3a0-17f6917c4615\") " pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.179306 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.179209    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/must-gather-r298n"
Nov 02 22:44:27.450368 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.450315    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/must-gather-r298n"]
Nov 02 22:44:27.454553 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:44:27.454504    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe3f80b_76b0_4690_b3a0_17f6917c4615.slice/crio-ab1767c8f8133843403edaefc610438e233c2c25ff205f100dece5dcab2a7f9d WatchSource:0}: Error finding container ab1767c8f8133843403edaefc610438e233c2c25ff205f100dece5dcab2a7f9d: Status 404 returned error can't find the container with id ab1767c8f8133843403edaefc610438e233c2c25ff205f100dece5dcab2a7f9d
Nov 02 22:44:27.456548 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.456516    4929 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:44:27.496309 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:44:27.496272    4929 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tdmzs" containerName="extractor"
Nov 02 22:44:27.676822 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:27.676787    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/must-gather-r298n" event={"ID":"1fe3f80b-76b0-4690-b3a0-17f6917c4615","Type":"ContainerStarted","Data":"ab1767c8f8133843403edaefc610438e233c2c25ff205f100dece5dcab2a7f9d"}
Nov 02 22:44:28.686064 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:28.686023    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/must-gather-r298n" event={"ID":"1fe3f80b-76b0-4690-b3a0-17f6917c4615","Type":"ContainerStarted","Data":"1d346fcde9692ccb48cf2640c77e3159dc0c60e1d4daea8782b34089338a6d70"}
Nov 02 22:44:28.686450 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:28.686070    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/must-gather-r298n" event={"ID":"1fe3f80b-76b0-4690-b3a0-17f6917c4615","Type":"ContainerStarted","Data":"45b8b215e900ac169c98333ad73c37da5a10fd0cb231674899c30c98b03b87bc"}
Nov 02 22:44:28.705179 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:28.703732    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7zc4v/must-gather-r298n" podStartSLOduration=1.764992265 podStartE2EDuration="2.703713573s" totalImagesPullingTime="938.721308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:28.701216626 +0000 UTC m=+2806.009307430" watchObservedRunningTime="2026-11-02 22:44:28.703713573 +0000 UTC m=+2806.011804379"
Nov 02 22:44:30.023724 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.023360    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bvq2f_9d39c020-6a26-47a2-8f57-1655acdc05f0/global-pull-secret-syncer/0.log"
Nov 02 22:44:30.148996 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.148959    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-plm4n_b103cd80-a503-475f-b9a8-f448fbd0af73/connection-checker/0.log"
Nov 02 22:44:30.174901 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.174871    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-q9m4v_0530c38c-3055-48b0-a829-ad3068c97528/connection-checker/0.log"
Nov 02 22:44:30.199400 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.199369    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-x8bf8_3a2a4927-992c-4f11-9f42-81405452f431/connection-checker/0.log"
Nov 02 22:44:30.305718 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.305634    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kzxpt_36174814-afd3-42b6-a555-75c832ebacf5/konnectivity-agent/0.log"
Nov 02 22:44:30.381855 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:30.381820    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-57.us-east-2.compute.internal_d522352ece329563a2383a55c8031694/haproxy/0.log"
Nov 02 22:44:34.671743 ip-10-0-131-57 kubenswrapper[4929]: E1102 22:44:34.671697    4929 upgradeaware.go:427] Error proxying data from client to backend: readfrom tcp 127.0.0.1:49166->127.0.0.1:37863: write tcp 127.0.0.1:49166->127.0.0.1:37863: write: broken pipe
Nov 02 22:44:37.669205 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:37.669175    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpmtr_e0ac2a3a-5b67-4a4f-a0b3-eafcd7d8dbf3/limitador/0.log"
Nov 02 22:44:39.193627 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:39.193595    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-g5w5w_a113cb77-4c64-47ea-8d09-723c2146038f/cluster-monitoring-operator/0.log"
Nov 02 22:44:39.483834 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:39.483729    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2cqw_c7914af0-6869-48fb-9b29-38c2da5b9dcd/node-exporter/0.log"
Nov 02 22:44:39.514555 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:39.514072    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2cqw_c7914af0-6869-48fb-9b29-38c2da5b9dcd/kube-rbac-proxy/0.log"
Nov 02 22:44:39.536887 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:39.536853    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2cqw_c7914af0-6869-48fb-9b29-38c2da5b9dcd/init-textfile/0.log"
Nov 02 22:44:40.042636 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.042602    4929 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"]
Nov 02 22:44:40.052102 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.052075    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.057016 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.056985    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"]
Nov 02 22:44:40.140559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.139188    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-proc\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.140559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.139334    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-sys\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.140559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.139512    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsflh\" (UniqueName: \"kubernetes.io/projected/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-kube-api-access-wsflh\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.140559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.139572    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-podres\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.140559 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.139620    4929 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-lib-modules\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240292    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsflh\" (UniqueName: \"kubernetes.io/projected/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-kube-api-access-wsflh\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240354    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-podres\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240400    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-lib-modules\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240477    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-proc\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240511    4929 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-sys\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.240697    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-sys\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.241142    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-podres\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.241224    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-lib-modules\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.242557 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.241270    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-proc\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.250394 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.250307    4929 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsflh\" (UniqueName: \"kubernetes.io/projected/6e899f5d-96f4-4948-86ea-c2aa3eff2f6f-kube-api-access-wsflh\") pod \"perf-node-gather-daemonset-zxwqt\" (UID: \"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.373554 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.373506    4929 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.664286 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.664182    4929 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"]
Nov 02 22:44:40.668415 ip-10-0-131-57 kubenswrapper[4929]: W1102 22:44:40.668375    4929 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e899f5d_96f4_4948_86ea_c2aa3eff2f6f.slice/crio-247eb3b4bf6bbef5ef5ebaa2938409709c0f10d0277fb0d735eece748f25fc9b WatchSource:0}: Error finding container 247eb3b4bf6bbef5ef5ebaa2938409709c0f10d0277fb0d735eece748f25fc9b: Status 404 returned error can't find the container with id 247eb3b4bf6bbef5ef5ebaa2938409709c0f10d0277fb0d735eece748f25fc9b
Nov 02 22:44:40.817838 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.817797    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt" event={"ID":"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f","Type":"ContainerStarted","Data":"e6059a7b4bf02adc108f881c1bc1166930421afe2627636f130f43c69ea50907"}
Nov 02 22:44:40.818037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.817866    4929 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt" event={"ID":"6e899f5d-96f4-4948-86ea-c2aa3eff2f6f","Type":"ContainerStarted","Data":"247eb3b4bf6bbef5ef5ebaa2938409709c0f10d0277fb0d735eece748f25fc9b"}
Nov 02 22:44:40.818037 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.818007    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:40.835367 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:40.835297    4929 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt" podStartSLOduration=0.835276437 podStartE2EDuration="835.276437ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:40.832963936 +0000 UTC m=+2818.141054742" watchObservedRunningTime="2026-11-02 22:44:40.835276437 +0000 UTC m=+2818.143367247"
Nov 02 22:44:41.228032 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:41.227997    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-vnfgs_7d36c5e6-09a7-4301-b6a1-db669e212d61/networking-console-plugin/0.log"
Nov 02 22:44:41.828354 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:41.828309    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/2.log"
Nov 02 22:44:41.834703 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:41.834676    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jpvb6_ec070a08-538c-46a0-8f70-bf23a1468b0c/console-operator/3.log"
Nov 02 22:44:43.032591 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:43.032560    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-grv5x_fbd984d3-01fe-4b43-9ac1-4259e6c39faa/volume-data-source-validator/0.log"
Nov 02 22:44:44.172197 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:44.172151    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kcsw9_feae0ada-a336-48c6-a163-7a968597ad33/dns/0.log"
Nov 02 22:44:44.195267 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:44.195230    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kcsw9_feae0ada-a336-48c6-a163-7a968597ad33/kube-rbac-proxy/0.log"
Nov 02 22:44:44.355401 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:44.355368    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcrch_bf18c217-668e-4f22-8bb3-ea403e33e962/dns-node-resolver/0.log"
Nov 02 22:44:44.964866 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:44.964830    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85npx_2afa6f34-35a6-4bcd-939e-8083b7cb4fb3/node-ca/0.log"
Nov 02 22:44:46.836186 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:46.836147    4929 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-zxwqt"
Nov 02 22:44:49.522666 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.522628    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-mtdsj_2cf18102-5f02-4e2c-b46b-043328bb8f66/kube-auth-proxy/0.log"
Nov 02 22:44:49.558889 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.558841    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-2zhcx_5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90/istio-proxy/0.log"
Nov 02 22:44:49.586699 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.586656    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-gjn6k_4a73ff87-2b7d-4c69-82e2-495f4b866345/istio-proxy/0.log"
Nov 02 22:44:49.622746 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.622709    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-j6cnb_047c4566-edc7-453d-bd8c-d14867259b93/istio-proxy/0.log"
Nov 02 22:44:49.744909 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.744878    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-wjhsd_e8cee094-9f9e-4b00-b658-431c6abdccc2/istio-proxy/0.log"
Nov 02 22:44:49.913435 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.913405    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:44:49.943561 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:49.943509    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8469b8f4f5-bf9r8_327d452e-cee7-493d-bd9a-81dc5b5b42e2/router/0.log"
Nov 02 22:44:51.815002 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:51.814964    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lknc7_c10347cb-927a-4d45-a48f-d9bc553d68db/serve-healthcheck-canary/0.log"
Nov 02 22:44:52.235661 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:52.235631    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-jx4w4_c61cf27a-32bb-4a64-9ef4-56badad04b6d/insights-operator/0.log"
Nov 02 22:44:52.236285 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:52.236262    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-jx4w4_c61cf27a-32bb-4a64-9ef4-56badad04b6d/insights-operator/1.log"
Nov 02 22:44:52.394869 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:52.394836    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdmzs_730c3bbf-87e5-4378-a258-a990d610143c/kube-rbac-proxy/0.log"
Nov 02 22:44:52.415365 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:52.415332    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdmzs_730c3bbf-87e5-4378-a258-a990d610143c/exporter/0.log"
Nov 02 22:44:52.437288 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:52.437258    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdmzs_730c3bbf-87e5-4378-a258-a990d610143c/extractor/0.log"
Nov 02 22:44:53.658570 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.658527    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-mtdsj_2cf18102-5f02-4e2c-b46b-043328bb8f66/kube-auth-proxy/0.log"
Nov 02 22:44:53.678725 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.678682    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-2zhcx_5e3f2d6a-0a59-4ad8-a517-4fc8ec0ade90/istio-proxy/0.log"
Nov 02 22:44:53.700012 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.699978    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-gjn6k_4a73ff87-2b7d-4c69-82e2-495f4b866345/istio-proxy/0.log"
Nov 02 22:44:53.716432 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.716397    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-j6cnb_047c4566-edc7-453d-bd8c-d14867259b93/istio-proxy/0.log"
Nov 02 22:44:53.789883 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.789851    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-wjhsd_e8cee094-9f9e-4b00-b658-431c6abdccc2/istio-proxy/0.log"
Nov 02 22:44:53.894087 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.894049    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9ggx_a5ffeb7d-4629-4724-9c97-a6aca4fbf01b/payload-processing/0.log"
Nov 02 22:44:53.915264 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:53.915178    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8469b8f4f5-bf9r8_327d452e-cee7-493d-bd9a-81dc5b5b42e2/router/0.log"
Nov 02 22:44:54.936525 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:54.936496    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c6f6f47c9-wjktb_94c958e5-ebd2-47a9-a69d-2eb5db98f6e4/manager/0.log"
Nov 02 22:44:54.957083 ip-10-0-131-57 kubenswrapper[4929]: I1102 22:44:54.957056    4929 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c6f6f47c9-wjktb_94c958e5-ebd2-47a9-a69d-2eb5db98f6e4/copy-manifests/0.log"